Wednesday, September 9, 2015

The Internship Experiment - Top 5 Challenging Scala and MongoDB concepts


In the recent internship that I had offered, I wanted the interns to experience new and emerging technologies that they were less likely encounter in their school's curriculum, and hence would add to the overall challenge. Scala, Play Framework, and MongoDB were chosen for this reason. To counter the steep learning curve, I kept the internship project goal simple. The interns were to develop a web-based survey creation and collection application, using the technologies mentioned earlier. In order to help the interns with learning these technologies, I shared the source code of another web-based application I created using these same technologies. We went over a code walk through so as to familiarize them with core concepts of Scala, Play, and MongoDB. As with learning any new technology, there were challenges. This article will list, and briefly describe, the top 5 challenges the interns ran into with learning and using these technologies for their project.

5. MongoDB: Joins and Aggregations

Schools teach RDBMS concepts, and they involve JOINing. Lots of it. Which is why NoSQL databases can be very confusing to understand at first. The interns were no different. The concept of rich documents excited them, and they seemed to believe that they understood the concepts well. They were also successful in creating CRUD code for simple documents. However, when they started work on slightly involved use cases, they began to miss their INNER JOINs and COUNTs from their Advance Databases Class. After a couple hours of revisiting document modeling in light of the use cases that triggered their SQL withdrawals, they became slightly more familiar and comfortable with using Rich Documents.

4. Play/Reactive MongoDB Driver: Writing code for asynchronous execution

Coding involves solving a problem, and so, we ought to code like we think. This involves anticipating and planning ahead of time, for possible decisions that we might encounter. Many of us relied on debugging and stepping through code, early on, as a learning tool. This, however, has an adverse consequence, as instead of teaching us to code like we think, it teaches us to code as we'd like the code to execute. This results in our code being written for sequential real-time execution. This makes it much more difficult to learn asynchronous programming, which involves coding for what might happen in the future, rather than in real time.

Things got even more complicated for the interns, when they had to write code involving multiple Futures, and tying their responses together, in a single code block. We spent a few hours revisiting this issue on more than a couple occasions, where we emphasized the need to "code like we think" rather than "code like we debug".

3. Scala: Implicit Conversions

Implicit type conversions is a very handy feature of the Scala language. It allows you to avoid writing intermediate type conversion code every time you have object of type A, but instead need object of type B. Writing an implicit conversion from type A to B gives you the ability to use object of type B anywhere you'd use object of type A. In my example code to the interns, I had provided them with an implicit conversion of a model case class to, and from, a BSONDocument type, so that I could pass the model case class directly to the MongoDB collection functions, instead of creating BSONDocuments for them every time. The interns soon forgot the concept and assumed that the MongoDB driver worked directly on the model classes. Many of their MongoDB driver related errors were, in fact, a result of copying-pasting code for implicit conversion from one model class to another.

2. MongoDB: Accessing the _id for an inserted document

MongoDB automatically creates _id key, and sets it to a new ObjectID, whenever you insert a document to a collection without an _id key. In the example code for a model class, I had also provided code for implicit conversion for the model case class to and from BSONDocument type. The implicit conversion to BSONDocument accounted for a missing id member of the model class. In that case, a new BSONObjectID was generated and assigned to the id member. This was part of the implicit conversion.

However, since the interns had forgotten all about the purpose of the implicit conversions, they were confused when they encountered a use case, where they had to use the id of a freshly inserted document, as a key in another document, in a different collection. They wanted to know how they could retrieve the id of a newly inserted document. The solution was simple. Supply the id yourself before inserting the document the first time. Since you control the generation of id in code, you could use it however you like.

1. Scala: Type Inference

Type inference is a very powerful feature of the Scala language and compiler. If a type can be inferred, chances are, you can get by without specifying the type of the object you are creating. This can, however, result in some unexpected behavior, and compile time errors, that could be confusing to interpret if not used carefully.

The most common complaint and frustration I heard from the interns had to do with this feature. The easiest way to troubleshoot type inference related issues happens to be to not let the compiler infer type of the problem code block. Explicitly specifying the type of the object created, or the return type of the code block, helps isolate the issue to the specific code block that might be causing the problem behavior. Once the problem is fixed, you can then omit the type information.


The above challenges were in no way a reflection on the interns' aptitude or ability to solve problems. I remember encountering similar challenges when I was first introduced to Scala, Play, and MongoDB driver for Scala. My academic training had not prepared me sufficiently for these challenges either. That was the primary reason why I chose these technologies for the internship. The interns admitted their frustration on my decision, but in the end were grateful for the same, as it pushed them out of their comfort zones, and gave them a safe platform to explore these technologies and concepts.

Monday, August 24, 2015

The Internship Experiment

The Internship

Ever since I started my independent software development consulting practice in October of 2014, I'd been planning towards offering internship opportunities to college students majoring in Computer Science and Business majors interested in Business Analyst and Software Development careers. Given the challenges new college graduates face in the workplace, due to lack of exposure to many tools, technologies, and development methodologies predominantly used in enterprises today for software development, the idea of offering internships with an emphasis in those areas appealed to me greatly. I also wanted the internship to be a platform where the interns would get a taste of some of the technologies that are not generally covered in their college curriculum. The internship would also serve as a test for my management and mentoring skills as this would be my first such experience.


I was able to reach out to college students through an acquaintance who worked closely with students in UT Austin, TX, and Texas A&M University, College Station, TX. Special care was taken in selecting interns and we selected students who were in good academic standing. We decided to offer three internships for IT Business Analyst position, and three internships for Software Developer position. The internship was designed to last six to eight weeks, where we met for couple hours each, two days a week.

Areas of Focus

The project was to create a web based anonymous survey creation and collection tool. The internship was designed to cover the following areas:
  1. Software development life cycle and methodologies
  2. Requirements gathering and documentation
  3. Use case documentation
  4. Requirements and Use Case traceability 
  5. Software Specifications documentation
  6. Software Specification and Use Case traceability
  7. Unit, Integration, and End-to-End testing
  8. Work division and delegation
  9. Source code version control 
  10. Timelines, Deadlines, and Accountability
  11. Software development tools and technologies
The software programmer interns were asked to use the following technologies for their internship project:
  1. Source Version Control using Git and Github
  2. Programming using Scala and Play Framework
  3. NoSql data persistence using MongoDB

The Experience

The internship went surprisingly well.While where were snags and unexpected issues, the overall experience was quite pleasant. The interns were eager to learn various aspects of Software Development Life Cycle. The quality of documentation produced for BRD and SDS was decent as well. The Business Analyst internship ended ahead of schedule. The Software Developer internship went into overtime and stayed behind schedule for a few weeks. While I was aware of the steep learning curve with Scala, I had not anticipated how much of a delay it would introduce in the overall timeline. We had an in-depth discussion on timeline, the importance of quality estimates, and the real world ramifications of missing deadlines. There were other issues with missed and misinterpreted requirements as well, which served as additional learning exercises.


At the end of the Software Development internship, we had a working product that was about eighty percent ready, and while it had a few rough edges, it was able to demonstrate nearly all the functionality and use cases that were described in the Business Requirements Document. The post-internship feedback from the interns was overwhelmingly positive and I received some constructive feedback on how the experience could be improved and enhanced for future interns. The purpose of the internship was to provide interns with a safe collaborative learning platform, to experience and participate in a Software Development Project, and to help them better prepare for the opportunities ahead of them. It also provided me with valuable lessons in software project and team management. Overall, it has been a very fulfilling and satisfying experience.

Friday, January 2, 2015

Android Tether - When the Linux Laptop won't connect to WiFi but the Phone does

Linux users and the age old WiFi problem

A Linux die-hard, who'd rather distro-hop, than run any non-Linux based OS. After spending hours trying to figure it out, managed to get the problematic WiFi module to work with the home WiFi router. The laptop connects reliably to the Internet. Life is bliss. Or it is, until the fateful visit with the laptop to a friends or relative's place, a hotel room, or a local coffee shop, that has graciously allowed its patrons to use the Internet connection over WiFi, and the Laptop refuses to connect to the WiFi.

For best impact read the above again, this time in Rod Sterling's voice, like the opening from "The Twilight Zone".

Those familiar with this scenario can feel the frustration one goes through in this situation. I've gone through it many times over the years, and recently, my wife had to as well. The biggest cause for this is that, majority of WiFi modules that are packaged with laptops these days, come with little or no manufacturer support under Linux. That is the unfortunate state of WiFi networking under Linux.

For situations like this, I started carrying with me a USB WiFi module known to work under Linux, as a fail-safe. I sacrifice bandwidth for reliability and it has proved useful over the years. The flip side of this was that, the Android phones I've used over the years, always connected successfully to almost any WiFi network, and I could reliably access Internet over them.

The irony of the situation was that, the solution to my problem was in my pocket all this time, if only I had cared to check it there.  

Android Phone comes to the rescue

Almost all modern Android phones natively provide multiple forms of tethering options: WiFi tethering, USB tethering, and Bluetooth tethering. The good news is, some of these options could come in handy when faced with the WiFi connectivity problems.

WiFi Tethering

WiFi Tethering allows you to share your phone's data connection (3g, 4g, LTE, etc.) with computers that are within range, over WiFi. It transforms your phone into a WiFi Hotspot. The problem with this approach is, you are burning through your phone's data, which doesn't come cheap. Many carriers also impose additional charges to even allow data sharing over WiFi.

USB Tethering

This one took me by surprise. I had used USB tethering before; to share my phone's data connection with my laptops. Setting it up used to be more involved, requiring running a few commands, but newer Android phones have made it seamless to the point, where, it only requires one to check a checkbox on their phone, to activate it. What I hadn't thought of, or tested, earlier was that I could use USB tethering to even share the network that the phone was accessing over its WiFi connection. You do need a data capable USB cable for this to work. You'll know if your USB cable is data capable, if your laptop identifies your phone, and the phone identifies that it is connected to a computer.

Using this option, you'll be able to access your phone's Internet connection over a USB cable connected to you laptop. The connection speed will vary depending on your laptop and the phone capabilities. I was able to get nearly 15Mbps downloads and around 5.5Mbps uploads using this approach with my Nexus 5 phone.

SpeedTest over USB

Bluetooth Tethering

Modern laptops are shipping with WiFi modules that also have a Bluetooth radio. This allows the users to share files with other paired Bluetooth devices, as well as, do some other fancy Bluetooth things. For instance, I can pair my phone with my laptop over Bluetooth, and then all of my phone's sound notifications and other media playing, can be heard on my laptop. Another thing you can do over Bluetooth is share the Internet connection.

Like the earlier two options, you can extend your phone's data connection to the paired device. You may incur additional charges, but it is good to know you have that as an option. You could also, like with the USB tethering option, share your phone's WiFi network connection over to the paired devices - in this case, your Bluetooth enabled laptop.

Don't expect phenomenal speeds. This is Bluetooth we are talking about. My TWC Internet connection clocks nearly 20Mbps over WiFi. But when tethered over my phone's Bluetooth, it drops to mere 0.6Mbps. It's still better than your laptop's WiFi if it doesn't connect at all.

SpeedTest over Bluetooth

Initial issues with Bluetooth Networking

One issue I ran into, while attempting to setup Bluetooth tethering, was that, while the laptop identified the phone's Bluetooth network, and I could transfer files back and forth over Bluetooth, I wasn't able to connect to the Bluetooth network on my phone. Network Manager would show "configuring network interface" message, but always failed to connect.

I later realized that somehow the default configuration of the Bluetooth pair on my phone was interfering with this. The paired connection on my phone, for the laptop, did not expose "Internet Connection Sharing", as an option. I only saw "Media audio" and "Contact sharing", and both were checked. When I unchecked "Media audio" and tried to connect to the Bluetooth network "Nexus 5 Network" from my laptop, I was able to connect to the network. Checking the paired connection's option again on my phone revealed the a new checkbox option, "Internet Connection Sharing", which was already checked.

The Verdict

If you are on the go, with a laptop containing an unreliable WiFi module, you do have options. As long as you have an Android phone (only tested with an Android phone) and a USB data cable or a Bluetooth module for you laptop, you'll be able to stay connected. Of the two, USB tethering offers superior connections speeds, but Bluetooth tether offers yet another fail-safe in case you don't have a data capable USB cable with you. While you may not be able to reliably stream videos from YouTube or Netflix over Bluetooth, as with everything else Linux, it is yet another option.

Friday, December 19, 2014

A 2-in-1 laptop that just works under Linux: Dell Inspiron 13 7000 Series i7347

For quite some time, I've felt the need for an x86 tablet computer that I could carry with me when I was away from my main workstation. This need became a necessity when I started travelling for business, and while my trusty Bonobo Extreme, from System76, was everything I ever needed for a computer to be, it was also a little too bulky and heavy, which made it almost impractical to operate in tight spaces like in a car - while not driving, of course - and in planes. I knew that I needed something that was modern with a tight form factor like that of a Tablet, yet full featured, unlike many Tablets. 

The ideal tablet computer needed to be cost effective and with known compatibility under Linux 32bit and 64bit. I had considered buying the Surface or the Transformer earlier, but since this was going to be a companion computer and not the primary one, and I needed offline storage to install all my software for when I was on the go, the Surface didn't have enough storage or compatibility to just justify the cost, and in addition to the lack of sufficient storage, the Transformer wasn't powerful enough for my needs.

I discovered Dell Inspiron 13 7000 Series i7347 while researching 2-in-1 laptops on Amazon. After spending time researching it, it seemed like it was created with my needs in mind. The additional $100 off the sticker price of the Dell's original $599 made it an even more attractive purchase I could not afford to miss. I was a purchase I did not regret making.
The laptop is very slick and professional looking, which works great, since I intend to take it with me on my business and family trips instead of my more powerful, yet bulky, 17 inch Bonobo Extreme Laptop/Workstation. The hinges on the Inspiron are very sturdy and the laptop easily folds into a tablet. The touchscreen is very responsive. With a 500 GB hard disk at my disposal, I am able to use all my office productivity applications, as well as, software development tools and related services, without any difficulty. Now granted that this model is not an i7 or even an i5, but neither is it an Atom or ARM based. It provides a full computing experience with enough horsepower for smoother operation and functioning.

Installing Linux on it was a breeze. Almost everything worked right out of the box, and the only hiccup under Linux was, the not-so-well supported Wireless/Bluetooth module with Broadcom chipset. However, with little research, I was able to find Intel's Dual Band AC 7260 module, with the proper form factor, that was compatible and well supported under Linux. A version of that module, 7265, is also listed in Dell's specifications document for this laptop as an option. I suppose, if I had bought this directly from Dell, I might have been able to choose the Intel over the Dell Wifi module.

I also made the right call, when I decided to upgrade the WLAN module myself. The laptop is very easy to upgrade manually. The HDD, the RAM, the battery, and the WLAN module, among other things, are easily accessible, once you unscrew the base lid. On their website, Dell provides a Service Repair guide for this model, which provided all the necessary steps needed to help with the hardware upgrade, and that has only boosted my confidence in this machine and it's quality.

I am very happy with the purchase and would recommend it to anyone who intends to run Linux on it.

The 2-in-1 can be purchased from Amazon using the link below.

Note: there are two form factors the Wifi module is available under. You are looking for the smaller of the two.

7260NGW Intel® Dual Band Wireless-AC 7260 802.11ac, Dual Band, 2x2 Wi-Fi + Bluetooth 4.0

The Wifi Module can be purchased from Amazon using the link below. 

Thank you for reading and I hope you enjoyed the article.

Monday, July 28, 2014

How I learned to program in Scala: The Nexus 5 experiment

Few months back I broke up my long term affair with Mono and I started looking into the the Play Framework from TypeSafe as a replacement stack.

Through Play Framework, I discovered Scala and the wonders of functional programming. My interest grew even deeper when I installed Scala on my laptop and tried some example code in the REPL tool (think Scala interactive shell).

However, I found very soon that I was not able to dedicate enough time during the week to learning Scala due to my full time day job doing .Net development.

Something had to be done.

I thought, I have this wonderfully powerful device on me all the time - my Nexus 5. Surely, I should be able to put it to use to help me spend more time with Scala. So began the experiment.

I rooted my Nexus 5, installed "Linux Deploy" from Google Play Store and installed a chroot Arch Linux environment. I also installed JuiceSSH client and a VNC viewer application. I already had "Terminal IDE" installed which comes with a great android keyboard geared towards software development on Android.

The last piece to the puzzle, I bought a $25 ANKER bluetooth keyboard compatible with my Nexus 5.

I fired up the Arch chroot on my phone, downloaded tmux and vim using pacman and oracle java 7 packages from AUR (some PKGBUILD tweeking was needed).

Then I installed TypeSafe Activator from AUR. I also downloaded vim plugins like vim-scala and CtrlP which proved to be of great help.

It worked!!!

Using the setup described above, I was able to not only learn how to program Scala, I am also able to write software using Play Framework. I can test the same on my phone's browser. The code is backed up on github and I can work on the same codebase from anywhere - as long as I have my Nexus 5 and data connection.

And after the recent KitKat update with Cast Screen - I am able to cast my phone's screen to my big screen HD TV over ChromeCast. This solves the issue of developing on the relatively small Nexus 5 screen.

This experience has made me appreciate VIM, TMUX, and the terminal in general at a whole new level. Even when on my laptop, I now find myself doing more and more coding in VIM instead of IntelliJ.

Think about it, you can now get a decent android box or even a phone, and with a setup similar to what I described, have an almost full fledged development environment. Rich graphical applications can be written in Scala using Swing (sigh) and possibly other graphical toolkits and can be tested within the Arch (or other Linux distro) chroot using VNC.

Side note about Oracle Java instead of OpenJDK: I went with Oracle Java instead of OpenJDK due to an "issue" with OpenJDK on arm chips. Its something to do with JVM running in "mixed mode".

Monday, March 17, 2014

CloudNotes - New project - using the Play Framework

I recently came across the Play Framework and decided to take it for a spin. Needless to say I was amazed. I've decided to use it instead of .Net/Mono on a new project I've recently started work on.
The project is called CloudNotes. The goal of the project is to create a web-based collaborative notes taking and sharing application that is accessible from all platforms. As mentioned above, it'll be built use the Play Framework. I've chosen MongoDB as the database backend for this. LessCssCoffeeScript, and jQuery, all of which come packaged as part of Play Framework, will be used extensively.

Currently, the application features:
  1. Create and share notes using a web interface
  2. Notes dimension and position on screen are stored along with notes - think web based collaborative brainstorming tool
  3. GoogleDocs like collaborative editing of notes, although not yet realtime.
Features, I hope to develop into the application in the coming weeks:
  1. User authentication
  2. Access Control List on Notes and entries
  3. Realtime interface with AJAX calls (currently causes full postbacks)
  4. Mobile and desktop clients
  5. Items reordering
  6. Checklist module
The source code is hosted on github and licensed under GPL v2.
Once complete, I'll probably host a demo on a DigitalOcean droplet.

DigitalOcean offers cloud hosting for as low as $5/month or $0.007/hr for dedicated server with:
  • 512MB Memory
  • 1 Core Processor
  • 20GB SSD Disk
  • 1TB Transfer
  • Additional bandwidth transfer is only 2¢ per GB

Monday, August 26, 2013

Update 1 - Progress so far - IrrlichtDotNet - .Net library for irrlicht that works under Linux

For original article, please go to this article.

After I made the declaration of starting this project on irrlicht forums, roxaz from the forums expressed interest in the project and immediately forked it on github. We bounced around ideas and while I won't yet merge all of his modifications into my repo, I did like many of this ideas and implemented them after tweaking them to fit in with my code. The ideas that I implemented from his repo are:
  1. folder reorganization
  2. a python script that refreshes the CS project file with all the SWIG generated cs files
  3. file to automate many initial steps
I also spent some time getting myself more familiar with SWIG typemaps and was able to map irr::core::stringc to System.String for almost all instances. These originally got translated to a SWIG generated generic wrapper which wasn't any useful unless you created a companion helper class in C++ to handle the translation both ways between irr::core::stringc and char *, which gets mapped correctly to System.String. Once all occurrences of irr::core::stringc are mapped to System.String, helper class won't be necessary any more. I will also attempt to deploy more typemaps to handle other such mappings in IrrlichtDotNet project.