Category Archives: Database

ELK intro and Elasticsearch lessons from production

ELK stands for ElasticSearch, Logstash and Kibana. I had become acquainted with this during MongoDB Day – Bangalore on 19 May 2014 by Susheel Zaveri’s excellent talk. So, I was overjoyed, when the Elasticsearch Meetup Bangalore’s First Meetup coincided with my trip on 27 Sep 2014. Elasticsearch has got an open, RESTful API that makes it easy to build applications on top of it. It can process both structured and unstructured data, so you can derive insights from log files to Tweets to plain old CSV files, all in near real-time. Best of all, you can ingest data from all these disparate sources easily into Logstash, then search and analyze across all of these types of data with Elasticsearch, visualizing the results using Kibana. This stack makes these insights available to anyone in an organization through Kibana’s dashboards, which are share-able and don’t require programming know-how to use effectively.

These features – plus many more – make the ELK stack so flexible that it meets the big data challenges of a wide variety of verticals. A major financial company uses the ELK stack to do anomaly detection and root out credit card fraud. Another one performs analytics and sentiment analysis across social media data. Yet another one detects hacking on their networks, and yet another for full-text search across e-commerce sites with billions of entries.

Suyog Rao starts the talk while Drew sitsThe meetup was held at SpringPeople Software Pvt Ltd, Sector 7, HSR Layout, Bengaluru, Karnataka.  It had 2 speakers: Suyog Rao, Vedang Manerikar. It was free of cost, but required registration in a Google Form. Suyog Rao (@suyograo) started with an introduction to ELK. He started describing ElasticSearch as a schema-free, REST and JSON document store. The salient points of his talk were:

 

 

 

 

 

  • The popularity of ElasticSearch can be gauged from the total number of downloads, which stands at 10M in last 2 years.
  • An Elastic Search cluster can contain multiple Indices(databases), which in turn contain multiple Types(tables). These types hold multiple Documents (rows), and each document has Properties(columns). [Terms in bracket are relational counterpart]
  • It uses replication for high availability and performance. For horizontal scalability, it uses sharding.
  • It supports:
    • Unstructured as well as Faceted, structured search
    • Enrichment and sorting
    • Pagination and Aggregation

IMG_2731

He covered Logstash and Kibana next.

  • Logstash is a ruby app, which runs on JVM.
  • It allows one to collect, parse, enrich and store logs and events.
  • Kibana allows one to have beautiful visualization on top of Elasticsearch index with zero code.
  • The new version makes use D3 library.

He showed a quick demo. Actually covered a lot of stuff in short time.

 

 

 

Vedang Manerikar (@vedang) works with Helpshift, a mobile CRM company based out of Pune and San Francisco. [It’s a company, which has unique hiring practices. Refer my earlier blogpost on Building Silicon Valley culture in India]

IMG_2734The customer-facing side of Helpshift product is a simple chat feature within the app using the Helpshift mobile SDK. The business-facing side is a complex agent dashboard that helps the agent in processing as many issues as quickly as possible. This business-facing side is built on top of Elasticsearch. He shared the following nuggets of wisdom with us:

  • Elasticsearch does not have a book on it, although it will soon be solved. There are good references and videos, but nothing structured like a book yet.
  • Don’t use Elasticsearch as a primary database. The data should first go into mysql, MongoDB or other transactional datastore.
  • Though ES allows one to have a mixed mode node with both meta data and data, it is best to separate master and data nodes.
  • For multi-tenant index like Helpshift’s usecase, an index per customer is not a good idea, but something based on the index size.
  • He said helpful steps about bulk loading like controlling replica count etc, but I did not catch it fully.
  • Rolling upgrade of ES is fraught with risks, so it is better to spin up new cluster and decommission old one. [This was contested by Suyog and Drew]
  • Benchmarking is hugely important and should be done at staging and development phase to prevent aches during production. He mentioned about a tool called Tsung, which helped them benchmark percolators. Percolators allowed live notifications of new issues.
  • During runtime, a lot of debugging can be done using cat API’s, so make use of them.
  • Tune JVM parameters, like allocate more memory for young generation.
  • ES uses Lucene under the hood, so some troubleshooting might require understanding its working as well
  • RTFM – Basically read manual carefully. Pay special attention to the unit, whether a particular number refers to ms or seconds.
  • Advanced ES users make use of filters to make complex views.
  • There were many others, but I guess we have to wait for the presentation to arrive.
Advertisements

Writeup on MongoDB Meetup at Jabong

IMG_1557Mongo Dilli (meetup url) held its meetup at Jabong on Aug 22, 2014. We started around 6:15 after initial introduction of participants. At least a quarter of them were using MongoDB in production, while few had just started looking at it. About half of them had not tried MongoDB yet, but were extremely interested in it.

The first talk was by the hosts at Jabong, Supreet Sethi and Apoorva Moghey. They had audaciously run MongoDB on Raspberry Pi running ARM processor. Since MongoDB runs on small Endian machines till MongoDB Inc fixes SERVER-1625, they had to use download a fork (github url) of MongoDB and compile it.

While they were presenting, I was frantically trying to finish up presentation. I tried rigging the raffle bucket, but it did not work, as I did not win it at the end just kidding :)!IMG_1558

After this, I started with my talk on Product Catalog: Retail Reference Architecture with MongoDB. After all, I was at Jabong, India’s leading e-tailer! On a serious note, the schema design in MongoDB due to its document structure is different compared with relational ER modeling, so I chose a sample domain to illustrate general points. I did spend quite few minutes on answering general and introductory questions on MongoDB, nosql; because 50% of audience was new to MongoDB and a few entirely fresh to nosql.

After this Anuvrat Prashar from product review portal, Zopper presented his journey of Python and MongoDB. It was really a pleasure to listen to the nerdy talk. Interestingly, he had ssh’ed to his box from his colleague’s over the internet, as his machine did not have a connector to the projector. His presentation was HTML5 and transitions were taking time after action, but things worked fine. We learnt a big deal about Python MongoDB driver and a few wrappers on it. The crawling produces semi-structured data, which is easily digestible by MongoDB. It would be a nightmare to do the same on a relational database.

IMG_1563The most important part was the drawing of raffles to announce 3 winners. The prizes were sponsored by Jabong. We had nice snacks and a great time networking with enthusiasts and users of MongoDB afterwards.

Mumbai MongoDB Meetup’s first session

Mumbai MGauravAtMumbaiMUG1ongoDB Meetup group started late last year and had it’s first meetup on Feb 8th, Saturday. The speaker, Anand George, a MongoDB and Node.JS professional gave an excellent introduction. A MEAN (MongoDB, Express.JS, AngularJS, Node.JS) user for past 2 years, he showed a presentation and then went on to show CRUD in front of audience in Mongo Shell. The audience had prior experience in relational databases, like postgres, nosql like neo4j as well big data technologies like Hadoop. There were folks from Ugam Solutions (analytics), IBM (SI and product), Wipro (SI), Open Solutions (now a part of Fiserv), Exa India etc etc. Later we were joined by Gaurav, VP, Engineering, ScaleArc (the sponsor of the event), who asked generic questions on nosql. We even touched upon git.

 

 

 

 

 

At the end, we had informal meeting and snacks. We discussed that we could meet every 2 months. We collected what are useful topics for different people.MUG1Snacks

 

MongoDB Afternoons in Delhi and Bangalore

MongoDB held its 1st set of events, 10 months after opening its offices in heart of Cybercity, Gurgaon.

An Afternoon in New Delhi

The event started with a welcome note by Rajnish Verma, Director Sales, MongoDB India.

I went next with a talk on Schema Design in Document NoSQL World discussing about Blog System.

Before tea break, Anil N from Techgene covered Pelica Migrator and Ashish Mittal, Daffodil Software showed ERP system, namely Applane. Latter went on to win MongoDB innovation award that evening!

Matias covered new features of MongoDB 2.6, released on April 8.

Nikhil Nayab, Cignex showed scaling using sharding using effective shard key selection emphasizing on benchmarking to collect empirical evidence rather than any other method.

Matias came back to demonstrate MongoDB Management Service (MMS).

Abhishek Tajpaul, from Intelligroup described his experiences during building of social media analytics.

Next up was Jabong’s usecase of MongoDB, before the innovation awards were announced.

An Afternoon in Bangalore

Next stop was Bangalore, where other Matias, Abhishek, Anil N., Uday Kumar (different speaker from Cignex) and I repeated our talks (Well, audience was different 😉 ). Susheel Zaveri, [24/7] talked with a lot of love for MongoDB about storing user behavior logs in MongoDB and its integration with ElasticSearch for beautiful charts for insights!

Rediff News Publishing’s use of MongoDB was described by Subbu.

After this, Livingtree won the innovation award! We had a gala night afterwards with audience.

MongoDB introduction talk at Dr Dobbs Conference Bangalore, Pune in April 2014

Dr Dobbs Conference held its maiden conference in India in Bangalore on 11 and 12 April. Organized by UBM, it saw good presentation on Hadoop, MongoDB, source code analysis etc. Yours truly presented an introduction to document database MongoDB. After giving a brief introduction to different nosql database, I went on to describe various aspects of MongoDB. I took a simple blog application example. I designed an ERD (entity-relationship design) circa RDBMS. I described the schema design in MongoDB’s document data model covering articles, tags, categories, users, comments and web metrics. Then, I illustrated it with Python code (inspired liberally from my awesome MongoDB colleagues). Please find the presentation below.

Do let me know your comments, feedback on it by leaving a comment below or emailing me.

NoSQL Benchmarking talk at CMGIndia Delhi regional meet

KV Narayanan, the guest speaker

KV Narayanan, the guest speaker on Webex

I presented at first Computer Measurement Group-India, Delhi Regional Event on November 29, 2013 at TCS Noida Sec 4 Office. Guest Speaker started talking about how Obama’s healthcare website could not handle the load because of poor planning and execution, Why measurement is really important? 2nd speaker gave good insights into Oracle performance. There were the following speakers:

  1. K.V.Narayanan, Vice President, Govt ISU, TCS, New Delhi, Guest Speaker
  2. Kapil Malhotra, Architect, Dell Software, New Delhi on Oracle Performance – Benchmarking, Tuning, Replication – Best Practices
  3. Prasoon Kumar, Solution Architect, MongoDB, Mumbai, Benchmarking of NoSQL Databases
  4. Ramakrishnan Raghu, Chief Architect, Passport Seva Project, TCS, New Delhi, Making Software Performance Engineering an Integral Part of SDLC – Practice & Experience
  5. Ramakrishna Alavala, HiPODS, IBM Labs, Bangalore, Proactive Performance Management in Production System

I prepared the following slides with which I delivered the talk.

NVP & PeepalSys bring together a panel of start-ups on building silicon valley culture in India

Nexus Venture Partners (NVP) is an early and growth stage VC fund looking at Indian start-ups going after global market esp. US as well as US companies looking to enter India. PeepalSys follows Y-Combinator model into talent recruitment space. So, when both of these put together an extremely relevant topic of bringing Silicon Valley culture into Indian companies with a panel of extremely successful technology leaders from NVP portfolio companies, it attracted a packed house in Sumant Mangalkar auditorium, MCCIA, ICC Towers, Pune.

Introduction

NVP managing director Jishnu Bhattacharjee (Linkedin profile, @b_jishnu) introduced the topic to us. He clarified that we should bring good habits of Silicon Valley culture into Indian companies and also supplement it with our own elements. Silicon Valley has proven to be a successful model for innovation with a culture fostering flexible hours, meritocracy, and rewards using stock option. NVP had made its first investment in a Pune company back in 2000 in Sigma, which worked out of Pune IT Park, Aundh. More recently, they have invested in Druva, started by ex-Veritas (later acquired by Symantec), which provides backup solutions. He introduced the panelists to us, which I will sprinkle as I describe each talk. It was followed by a presentation from all 5 panelists of 3-minutes each, before Q&A panel discussion started between Jishnu, audience on one side and the panelists on the other side.

Presentation

Abinash Tripathy from HelpShift recruits SpecialOps guy

FoosballAtHelpshiftHelpshift makes it easy to create that exceptional customer support experience in native mobile apps, transforming customers into the most passionate sales team. Abinash Tripathy (@abinashtripathy)is the founder and CEO. He led another company called Infinity Beta, a thinktank of startup ideas like Paisa.Com. He recalled that they began talking about company culture even before they began working on product, wow! He illustrated the culture of city with a picture of Pune landmarks, festival, food etc. Culture is value system of a group of individuals, in this case startup employees. He contrasted an army vs special operations team. Whereas an army has the following characteristics: a large number of individuals, hierarchy, obedience, defined roles, and uses mandated tools; the special ops team has high-potential soldiers, only a handful of them, who would use innovative tools, will generally possess high IQ and make their own decisions towards a mission. Building such a culture starts with recruiting high-potential employees. Helpshift values attitude rather than skills. They have inverted the traditional recruitment model by making it inbound. So, instead of HR or external recruitment team reaching out to many engineers, the company pages and its work act as a magnet for the kind of talent, they want. It actually scares engineers with faint heart. This way instead of spending 2 valuable days just filtering out numerous resumes, Abinash and BG are able to focus on high probable. Paisa.com did not have a careers page. Instead it was extremely well architect-ed, so an inquisitive and interested soul would examine it and find it in some HTML tag an email for this purpose. And, that would be just start of a 3-months long recruitment cycle. Everyone was surprised including the host Jishnu, when he mentioned that. He then went on to describe kinds of questions and tasks in the interview process. They look for people, who can learn new skills, they have never used before, quickly come up to speed and accomplish a fairly non-trivial task with it. Prior experience (existing skills) don’t matter to them. One such example is: Using Arduino to build a Build Process Signal, innovative, very, very innovative. So, the candidate learns its device driver and integrates continuous integration system like Jenkins with it. Please read this comment made by AB in his blog AB’s (the blog is now private) answer on Quora for “How did Helpshift hire talented developers” for unadulterated version.

If one were to walk into Helpshift’s office, it’d look like a playground with foosball, video-games, mini-gym within the office. Some objects, which all of them possessed included:

  1. Amazon Kindle: All Helpshifters are avid readers. In this digital age, when many professionals I know have moved on just snacking on Twitter, blogs for keeping up with new trends, learning new professional and life skills, it is refreshing to know that company encourages such habits among employees.
  2. Emacs: With a plethora IDEs and WYSIWG editors, plain text editors like VI( M) and emacs rule geeks like Helpshifters. I remember meeting a Symbiosis Design student working at InfinityBeta bemoaning this fact Winking smile, that he wasn’t allowed Dreamweaver or something, rather had to use emacs!
  3. Clojure: Geeks at Helpshift love functional language running atop JVM.
  4. Raspberry Pi: DIY kit for making your own computer.
  5. Bitcoin: World’s alternative currency
  6. Special interest outside of work: Be it Maths, Guitar, robotics or Mahabharata, you have got to be of high calibre in one of your hobbies, that you pursue outside of work.
  7. Continuous learning and technology eco-system involvement: [Self observation] You’d find Helpshifters at Coursera, local Emacs group etc.

Their role models included Steve jobs, Richard Stallman and 7 more, which audience could barely identify. You will find such quotes like: Real artists ship.

A software engineer at Helpshift is expected to be self-didactic, whose definition none from audience was able to specify. In jest, I say that AB inverted panel discussion by making audiences answer, while he put forth questions! I’d say the audience including me were lousy by his high standards. Coming back to the culture, at Helpshift, the software developers also act as customer support in addition to product development. This way they are aware how their code performs in the field. Recruiting and appreciating employees is quite important in such cases. Otherwise, as noted in The Five Languages of Appreciation:

  1. Lack of praise and recognition leads to low job satisfaction
  2. Lower job satisfaction leads to higher turnover
  3. Higher turnover has a negative impact on customer satisfaction.

Jonah from Indix builds a team of ants for heavy data lifting

03Jonah Stephen Jeremiah (Linkedin profile) said the mission of Indix is to store a vast amount of data and make sense of it, derive interesting inferences from it. They have over billion product prices database and a market intelligence system on top of it. They are 35-member team and have an office ready to accommodate 35 more. In parallel with the metaphor of special ops, Indix has ants, who do heavy lifting well beyond their capacities. Indix fosters transparency right from office setup having no opaque walls for managers, even meeting rooms. He did not describe his recruitment funnel from start, but somebody in advance stage spends a full day in the office before (s) he joins.

Gaurav from ScaleArc impresses candidates by its work

Gaurav took the talks further, but the panel started digressing from Silicon Valley culture to company work and their recruitment efforts (We, audience also helped this change of course, but the discussions were stimulating nonetheless, so no big regret from my side). He started by describing how external circumstances like family and society view startups in India. He described how his dad consoled him on his startup job with words: You will get a a job in big company soon! Many marriageable aged boys and girls would take up an MNC jobs to boost their attractiveness to in-laws and potential partners. <Self observation> Many fresh graduates themselves seem to take a startup job as last resort after having been rejected by MNC, as a stepping stone to MNC. They screw up the startup and don’t give their 100% to their current employer. Instead prepare for interviews or MBA’s. On the other side, many MNC engineers will tell startups that they need higher salaries, because they are compromising on the brand of their company by coming to your startup </Self Observation>. The way ScaleArc would approach recruitment would be making founder accessible to the potential candidate, describing the great work they are doing, which the candidate should be able to fathom being technically adept herself/himself. Then, they would talk about the kind of clients using it already – likes of Flipkart, Microsoft, Kixeye (Zynga’s competitor), which would put any arguments to rest.

Ankit Pruthi from Unicommerce looks for ethical hacker

Unicommerce looks for ethical hackers, who is an equivalent of 100 programmers. With a team of 4 developers, they are managing a large scale SAAS order fulfillment system, being used by 1000 concurrent users, for 40 warehouses to ship 50K products everyday across India (Snapdeal, Jabong), SouthEast Asia (Lazada), even Pakistan and Canada. Ankit, the youngest panellist at 26 years of age talked about the custom scripting language they have created to help customers come up to speed quickly on their product.

Atul Phadnis of WhatsOn India builds a cross-functional team without silos

WhatsOn was ably presented by Atul Phadnis as a company, who are further along in their startup journey than rest of panelists. That POV (point of view) gave a glimpse of what you’d do, when you go beyond 30-70’s in employee strength. He was visibly proud of having hired best talents from large companies, two in recent past from Airtel and another one, I don’t remember. He spoke of the high energy atmosphere within the company, which was observed by potential recruits to be a reason for the same. I really liked the fact that he democratically chose an office inside or very near mall in Worli, which many companies are shy of doing lest their employees get distracted. He spoke of a road-show in bay area, where WhatsOn showcased their technical prowess to equally competent silicon Valley crowd and got nod for good work. He also talked about expanding his company to South East (Indonesia) and Middle East (Jordan) Asia and integrating them into WhatsOn culture, very mature POV coming from him.

Q & A

It became more interactive towards latter half by design. I’d leave that for a new blog post in future.

Mobile and Social enterprise

Taking your Enterprise Mobile is no longer an option, but a necessity

Mobile adoption is increasing in industries – healthcare, construction, govt. Field workers can download a part of workflow offline. When reconnected, it’d synchronize with the backend application. CIO

  • 77% plan to use personal devices to access company data and applications.
  • Deploy more than 25 mobility apps in next 2 years.

Mobile enterprise strategy

  1. Build and Connect: Connect to and run backend systems in support of mobile
  2. Manage and Secure: Mobile Devices, services and applications
  3. Extend and Transform: Existing business capabilities need to be extended to mobile devices. Also, create new opportunities to transform the business

Technology backbone

The priority needs to be decided for targeting mobile OS. If it is primarily smartphone devices, a technology like Phonegap can be used to reuse code across iOS, Android, Blackberry and Windows phone 8 OS. The technology should be able to support:

  • Social networking,
  • smarter commerce, and
  • analytics.

Sometimes you need to look at the support for feature phones in addition to smartphones. The analytics piece mentioned above, allows developer to track the mobile experience. It also means automatically detect customer struggles, obstacles, or issues. It should enable us to drill down into actual user behavior.

Business Continuity with unreliable networks

The solution should enable companies to connect a distributed system (like point of sales or the sales force’s mobile devices) to a centralized database even though the network infrastructure is unreliable, a common challenge in India. The solution also should provide a cost-effective way to automate highly distributed business processes leveraging remote knowledge workers by replacing heavy web/app servers with lighter client apps.

Build and Connect innovative apps and interactive experiences

Background: There are going to be 10B devices by 2020. 61% CIOs put priority on mobile 45% say that they can achieve increased productivity with mobile apps according to a report by IBM.

It will find its way B2E, B2C applications in different industries like – Retail, Financial services, Distribution according to 2011 IBM tech trends report. It also mentions top mobile adoption concerns. Increasing trend that we are going to witness is BYOD (Bring your Own Device). There will be 200m BYOD soon. My employer Justdial has rolled out Ambassador program, which empowers employees to take our enterprise software on Ipad with 3G connection, wireless Logitec keyboard.

The increasing number of operating systems as well as the mix of devices, form factors and standards has turned native application development into a cost-prohibitive strategy, resulting in a growing market of cross-platform solutions and mobile development tools, like Phonegap and IBM Worklight. It provides device application programming interface (API) using JavaScript.  So, one can use HTML5, CSS4 and JavaSscript libraries like dojox.mobile, Sencha Touch, jQuery Mobile. There a provision for Server – push notification, version management, security, integration. APN (Apple Push Server), Google Push Servers (GCN, formerly C2DM), RIM Push Server, Microsoft Push Server

Social Communication

Focus for enterprise unified communication has shifted from reach to relevance. Messaging tools need to provide real-time translation across languages, and IM lingo as well. Soft phones are in vogue, as they use broadband removing voice costs and have better integration on applications. This can be extended to customer interaction as well. The following are components:

  1. Voice
  2. Meetings
  3. Video
  4. Mobile
  5. Instant Messaging
  6. Social
  7. Call Management

There are going to be 1b smartphones by 2016. Do you need more reasons to focus on enterprise mobility.

SAP, Database, ERP et al

Keynote

Asian paint case study – 5k employees, 30k shops. Android platform. ERP platform.

Successfactors – As big and as old a SAAS company as Salesforce. 15m users. 3500 customers >10 yrs in cloud.

Indus SAP user group – Manish Choksi, CIO Asian Paints Ltd. is the chairman. SIG on BI, industry auto, pharma launching soon, platform. Partner summit.

Steve Lucas – Executive VP & GM, DB & Technology. sap.com/rds Rapid deployment Service improves time for SAP deployment.

Paradigm shift: 85B apps in 2016. 8B population. Big data, real time, predictive. 7.5 ZB 90% video.

SAP visual intelligence has been inspired by Tablaeu in an open admission by Steve. Explore it on microsite.

Social mobile panel

Niraj Athalye. Neelam Dhawan (HP)

Sentiment analysis, intention. Asian Painting will help consumer find the right painting contractor with the help of friends. People as media – socialcasting. Personalization of software.

Analytics

Decision revolution – inform, align, adapt.

BI, EPM, GRC

  • Query & reporting
  • Dashboards
  • Data Discovery
  • Predictive analysis

3d model from right hemisphere acquisition integrated into analytics. @atul_sap

Ashok leyland

Venkatesh. Building information culture by leveraging BI. Sap ECC, dealer business management. SAP is used in all joint ventures for collaboration and monitoring as well. Business analytics is a subset of BI. SIAM competitive data, and Commodity price of steel, benchmark. Business insights. IT maturity: supplier, enabled, IT led organization. Hubble telescope into business operations and performance.

Information maturity path 

  1. Data analysis
  2. Information
  3. Performance
  4. Predictive

BA maturity

Data mashup

Access control

BI obesity

VmWare

Infrastructure efficiency, simpler management,  built-in availability, greater agility

NIC HBA teaming

Web application framework

Product Front End Back end Database Others
Twitter Ruby on Rails (RoR), JavaScript, jQuery
LabJS, Modernizr, JSON-P, oEmbed
Scala Cassandra Java, C, Python, Mustache templating language
Facebook PHP, XHP, Hiphop for PHP, JavaScript C, C++, Java Cassandra, MySQL Python, Erlang
LinkedIn JSP, Apache Coyote Web Server Spring MVC, Linkedin spring, grails, Oracle and MySQL ActiveMQ for JMS, Lucene as a foundation for search, DWR, Jetty, Eh-cache, Quartz, Spring remoting.
YahooMail HTML, CSS, JavaScript (with YUI 3) PHP MySQL Apache Traffic Server (formely known as Yahoo! Traffic Server).
Google + Closure framework, including Closure’s JavaScript compiler and template system, HTML5 History API Closure templates server-side, C++, Java, or Python BigTable and Colossus/GFS MapReduce
FourSquare scala(lift framework) scalaAmazon S3 for hosting, /img/ folder which is served by nginx directly MongoDB load balancer(s): nginx/0.8.52Lift- A web framework written in scala.
Youtube Python psyco, a dynamic python->C compiler MySQL
Quora Python and JavaScript LiveNode/webnode2, Thrift (Communicate to backend)Amazon EC2 and S3 for hosting MySQL + memcached C++
Load Balancing: nginx in front of HAProxy
Viddler PHP, Python Rails 2.x, ffmpeg/mencoder/x264lib, Java 1.6 / Spring / Hibernate / ehcache, ErlangAmazon EC2 and S3 for hosting Mysql 5.0 Hadoop HDFS (distributed video source storage)
Nginx/Keepalived (Load balancers for all web traffic)
Wowza (RTMP video recording server)
Mediainfo(reporting video metadata)
Yamdi (metadata injector for flash videos)
Puppet(configuration management)
Logcheck(log scanning)
Duplicity(backup)
StackOverFlow jQuery, ASP .NET C#, Microsoft ASP.NET (version 4.0), ASP.NET MVC 3, Razor. LINQ to SQL, some raw SQL HAProxy (for load balancing), Bacula(for backup), Redis(caching layer)
Disqus jQuery,EasyXDM, Sammy, Flot, Raphaël, JSHint Python scripts, Django, Celery, South PostgreSQL, memcached HAProxy + heartbeat (Load balancing)

In Short..

Database Distribution

Backend Technology Distribution

%d bloggers like this: