[I wish to see] Scope of social search

Social search technology is still in nascent stage. Big companies(well in terms of users) like Facebook, Google have introduced semantics in their search but still lot to improve.

Let me list out the examples to show where we are

If I am a photographer, I would ask in search to show me the scenic photos which have more than 1 million likes
If I am a policeman, I would search who violated rules
For example, a person tagged in a location A and location B and their difference in time < expected time of travel. 90% chance of violation of traffic rules.

In terms of content classification, here are few examples:
Videos:
Search in a video: show the scenes between two actors/people
Show me all videos of an actress
Show me all action scenes of a video?

Photos:
Show me all friends who are brighter than me?
Show me all friends who visited Nayagara falls? . (Friends may have just uploaded photos without Nayagara falls description. This information has to be derived from photos)
Show me all photos which are not liked by 10% of my friends

People:
I want to search for people who are inactive since?
I want to search for people who I haven't talked to me since?

Places:
List the places I may like to travel? (this can be derived from friends's visited places and similar places that I visited earlier or photos I like)
List the places to have a dining? (same as above)


Having a 100% faith on social search results is wrong because many of our friends may not have updated their fields/information.

I don't care about above examples except video category. I wish to develop these search features.

Projects related to this area: Google search bar can be included in any external sites. Similarly, one can implement a semantic Graph search bar.

Confusing terms

You are not confused if you are able to answer following questions without any doubts

Q) Can we build a system which is highly fault-tolerant but not highly available?
Q) Can we build a system with low reliability and High availability?
Q) is reliability defined as MTBF or MTTR?


Scalability
A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system.

Elasticity
Elasticity often refers to a system's ability to allocate additional resources in an autonomic manner

In other words, a scalable system allows you to add resources in order to handle more load, while an elastic system will add resources itself when the load increases.

Fault-tolerance

Fault tolerance refers to a system's ability to continue operating, perhaps gracefully degrading in performance when components of the system fail. There is no exact measure to measure fault-tolerance of a system

Availability

Availability is a percentage of time that a system is actually operational and providing its intended service.

A = Uptime/(Uptime + Downtime)
Ai = MTBF/(MTBF+MTTR)

Where there are no single points of failure might be considered system as fault tolerant, but if application-level data migrations, software upgrades, or configuration changes take an hour or more of downtime to complete, then the system is not highly available.

Reliability
In simple words, how long can a system stay up continuously?

More concrete definition is “reliability is the ability of a person or system to perform and maintain its functions in routine circumstances, as well as hostile or unexpected circumstances” 

Reliability is often defined in terms of mean time between failures (MTBF). We can build a system with low-quality, not-so-reliable components and subsystems, and still achieve HA.

Durability
Durability of a system guarantees that stored data can't be lost.

References:

1) http://www.quora.com/Distributed-Systems/What-is-the-difference-between-the-terms-scalable-and-elastic
2) http://www.ibm.com/developerworks/library/pa-bigiron2/


3) http://www.quora.com/Distributed-Systems/What-is-the-difference-between-a-highly-fault-tolerant-and-a-highly-available-system#
4) http://www.wikipedia.org/

Pregel:Google's way of processing graphs

Introduction
Many applications can be modeled as graphs. Google has introduced the Pregel to process large scale graph processing applications. Lets delve more into this Pregel framework.

Few inspirations towards graph processing frameworks:
1) Facebook’s social graph contains 721 million users, 69 billion friendship links. Average distance between two users are 4.74
2) Representation of Wikipedia articles in a graph
3) Database trend is moving towards Graph Database.

Pros:
Large scale graph processing distributed system
Provides fault-tolerance capabilities through checkpointing
Performance of system is improved by bulk synchronous computation
API is modeled as a ‘think like a vertex’

Overall, Pregel has influenced state-of-art towards graphs.

Cons:
Lets talk about issues with Pregel using an example.
In one case,micro data centers are distributed geographically. Nodes in a data center are heterogeneous in nature.
In other case, graph is processed in homogenous mega data center. Performance of Pregel(considering a design in paper) in case 2 is much better than case 1 for following reasons.
1) Pregel doesn’t consider heterogeneous nature of system while partitioning the data. Slow node gets same size of data but can’t perform in similar speed as a fast node.
2) Due to single synchronization barrier, slow node slows down entire whole graph computation.
3) What if Pregel partitions graph data such that there involved huge communication between nodes which are far away.
4) Shape of graph is not considered in partitioning data.

Current state:
Twitter has introduced Cassovary, another big graph processing library. Many projects has inspired from Pregel. Open source projects are Apache Hama and Giraph. Giraph is has strong contributors from Twitter, Facebook, LinkedIn.

Brief descripion of inspired projects:

Apache Hama	Pure BSP implementation over Hadoop
Giraph	It is almost similar to Hama
HipG	Java based library and no single synchronization barrier
Signal/Collect	gives same importance to vertices & edges instead of focusing on vertex.
Phoebus	Pregel in Erlang

Discussion Points:
MapReduce vs Pregel
Issues with single synchronization barrier
Best way of partitioning graph data

Importance of communication

My biggest weekness is lack of communication skills. Hmm... I am not sure. May be it is because so many people told me. It is more difficult get over any weekness when we constantly think about it.

Lets go back and check what I have lost of because of lack of communication skills

- Had a big fight with my best friend because of mis-communication

- Couldn't perform well in HR round of Microsoft in undergrad

- Technical skills were not valued at D.E.Shaw because of lack of good communication skills.

Well, Let me think again. 

I do work hard and I am good at thinking. 

what if I read many grammer English books.
- I did. But it didn't help. 

what if I read many novels?
- Technical books are more interesting than novels. I tried and read around 15-20 books.

what if I watch so many English movies?
- This thought changed my life. I started watching one movie per day. Now, I am unable to get rid of this bad habit.

what if I watch one movie many times?
- It helps me to remember dialogues. 



what if I read one novel many times?
- I read 'Two states' novel 3 times.  can't read.


what if I write a novel?
I wrote a novel 'Indian bridegroom' and published as well at https://www.smashwords.com/books/view/125514
You know what. I was appreciated for my narrative skills. Yes, this novel writing helped me to impress sentence patterns in my mind. 

There is no single way to get over any weekness. Constant aspiration being excel in one area helps a lot.

Work in the boundaries of capabilities/knowledge

Its very difficult to quantify the one's capability in a area. And it's very certain that there is always an improvement. Why do we need to think about this?. World is so competitive. When you want to be top, want to prosper in life,(s)he has to know about their area of improvement.

Let me give a simple example to emphasize advantages of working on boundaries of knowledge. When we read a fresh topic, we get lot of new thoughts. These new thoughts might have been well explored. But if we get a new thought after having complete knowledge about a topic leads to a research paper. 

Every day is same

You believe it or not. Every day is same. Atleast, almost same.

Sun rises around 7AM and sets around 6PM. In a working day, the people we meet are also same. 

Wake up in the morning. Get ready for office. Do some stuff at office (this is NOT to save country but just to lead life). Come back home and then sleep.

Routine life makes our boring and doesn't bring a smile.  What can we do? 

Should we stop following a routine? All professionals are not same. We can't do this in every job.

Though our lives are same everyday, our thoughts are different. You can do the same thing differently and in a better way. Everyone is different. Be passionate in knowing other's life. Relationships grow. Every person is an unexplored book.

Dance @ Deshaw parties...

We (D.E.Shaw treasury) went to excess pub, which is in Novtel 1st floor. I danced a lot, may be this is because my mind is fully occupied with Michel Jackson inspiration. Prasanna, my colleague, dearest friend at Deshaw. He danced exactly like a dog. Almost everybody in my team danced except my team photographers, the only and one subbu, and Sravanthi. You can very well comment my dance. Here is youtube link http://www.youtube.com/watch?v=DIugQdeEFdA
If you compare the dance I did in Diwali occasion, this dance is much better.

Anubhav and Ujjwal talks after they drunk are bit funny. Ujjwal want to make Deshaw as one of greatest hedge fund and optimizes all calculators in order of O(1). Anubhav is taking care of mastiff up and running. We had a chat with our M.D, Neal Power, the big boss. It is a different experience to talk with highly influential persons.