Lack of Embedded System Jobs?
I recently wrote a couple of think pieces for IEEE and Embedded Intel Magazine discussing the sad state of affairs regarding the dwindling pool of embedded systems developers. Basically, my thesis was that many institutes of higher learning are no longer teaching CS majors what computers are all about from a low-level perspective (CPUs, caches, data busses and the like) and this is causing a shortage of embedded systems developers. This thesis has been backed up by folks like Jack Ganssle and the many of email responses that I have received from all over the world. Evidently, it's not just a U.S. problem, but a world-wide phenomena.
Nonetheless, I've also received a few emails stating that if there were jobs, then there'd be more demand and therefore more "bare-metal" engineers to fill the positions. That thought got me wondering. I believe that the reason that there are apparently no jobs is because the embedded systems development space is a stealth industry.
We inherently know that embedded systems are almost everywhere. However, as seen at ESC San Jose in the keynote presentation, not a one of the "man on the street" interviewees could identify just what an embedded system was, let alone identify one. The same applies to determining the size of the embedded job market. If you don't know what and embedded system is, how do you know what the jobs are?
The stealthiness of embedded systems is both good and bad. On the plus side, it means that embedded developers are doing their job correctly. On the minus side, no one can identify just what the job was -- let alone what skills it required and how to train/educate for it.
Now, I define an embedded system as one wherein you inherently know there must be a computer in the device someplace, but you're just not sure where. By being a well-designed embedded system, the user is oblivious to its existence. This makes it extremely difficult to define the scope of the embedded systems market precisely because we're doing such a good job at hiding computers everywhere.
As to whether or not there are jobs in embedded systems here in the U.S., I went to Monster and had a look. A quick search showed about 300 job openings for RTOS developers. Close to 1000 for firmware developers. 2,500 for embedded developers. Over 5,000 for systems engineers. 678 for embedded Linux developers, etc. Now, admittedly, there will no doubt be some overlap where there may be duplicate hits for the same opening. But, to say that there are no embedded systems jobs here in the U.S. is just plain wrong.
However, the problem is that in order to understand the scope of the job openings, you need to understand where embedded systems are used. This is one of the fundamental failings of the university system when it comes to designing their curricula. Since the academic community is seemingly no better at identifying embedded systems than the "man on the street", they woefully underestimate the demand for "close to the metal" understanding of computers.
And, as for those who say that all of the embedded jobs have moved to India and China, I'd say that that's a load of crap. Yes, there is a lot of embedded development going on in India and China. But, to say that there's no development left here in the U.S. is just ignorant of the real situation. If that was the case, then there wouldn't be representatives from large high-tech firms lobbying Congress for 195,000 H1B visas for "highly-skilled technical positions" for 2008. Now clearly, most of those H1Bs are not targeted for embedded developers. But, some of them are. And, those openings could represent several thousand jobs going unfilled.
Now, I *do* believe that there's a problem with the salaries that those companies are willing to pay. And, I feel that the proliferation of H1B visas helps depress the salary levels of developers which precludes otherwise promising students from pursuing careers in embedded development. That will be the topic of a future post though.
As Al Gore said at the 2007 ESC keynote, the future of the world and our ability to address the issues of global warming and energy conservation rests in the hands of the embedded developers of the world. Without embedded systems, none of the fancy technology in things like hybrid cars would ever work. So, we need more clever people now than ever before. And, we need to get the educators and the legislators involved to help address the misconceptions about the death of our industry.
So, to those of you who may be thinking of a career in embedded systems development, whether you're a CS, CE, EE or just someone who likes to make things work, don't be discouraged when the naysayers tell you there is no future in computers here in the U.S. The U.S. used to be the technical innovators in the world. We may have lost our edge, but the work is still there and we're still doing innovative stuff. But, we need you to participate.
But, that's just my opinion. What's yours?
Lack of Embedded Support for Debian-based Distros
Greetings!
I've noticed something over the past few months. Namely, none of the commercial distributions of embedded Linux support Debian-based distributions for development. Neither WindRiver, MontaVista, LynuxWorks, nor Timesys support any Debian-based installations. This rules out easily installing any of their commercial products on distributions like Ubuntu. Yes, I know that you can use alien. But, many of the commercial packages will simply refuse to install on non-rpm distributions. This limits you to SuSE and RedHat for the most part, and neither distribution is pushing the community like the Ubuntu folks as far as integration of hardware and new features.
Don't get me wrong. I'm glad that these vendors support Linux at all. I'm just disappointed that they are not supporting what is arguably the most popular Linux distribution in the world. I believe that this failing is keeping them out of colleges and universities where students and their institutions tend to favor the Ubuntu distribution. So, let's get with the program guys! It's time to realize that RedHat and SuSE do not constitute the bulk of development platforms and at least support one Debian-based distro.
Multi-Core: Solution or Problem?
We hear a lot of noise from the silicon manufacturers today about how multi-core processors are the wave of the future. Certainly, they do allow us to pack more performance into a single package. However, are they a good fit for embedded systems?
One argument is that with two or more cores at a lower voltage, we can run with a lower thermal dissipated power envelope. This is certainly true. Power consumption can certainly be lower in certain applications. And, with multiple processors, the potential is certainly there for doing more work with less heat and power consumption.
However, although multiple cores can make good sense from a hardware perspective, they can be a real problem from a software perspective. There are basically two approaches for operating systems in a multi-core environment. One is to partition the cores and run two separate operating systems. This can allow you to mix OS variants so you could have say, an RTOS on one core and a something like Linux on the other for the user interface. Naturally, this complicates the communications between the cores and requires partitioning the memory and assigning certain devices such as PICs to one OS or the other. Essentially, this becomes two uniprocessors in the same physical package.
The other approach is to run an symmetric multi-processing (SMP)-aware operating system. Unfortunately, there aren't too many of these on the market just yet. VxWorks, Linux, QNX and maybe a couple of others. Running an SMP-aware OS can simplify some things such as load balancing. Interrupts and processes/threads can migrate from one core to another based on work load. This can be good. But, it can also be very bad.
The problem comes in when, say, two threads that would have normally run A then B on a uniprocessor due to priority preemption, suddenly run simultaneously on both cores. The potential for race conditions is rampant. We must make sure that our code is not only logically correct, but
temporally correct as well.
Consequently, running on multiple cores requires a significant amount of thought on the part of the software designer. The use of synchronization primitives to ensure proper sequencing is imperative. This is especially true for operating systems like Linux that can schedule different threads from the same process on different cores simultaneously.
In order for us to get the most from a multi-core system, we need to take advantage of the blocking side-effects of certain primitives such as message queues and semaphores to make sure that we run not only the correct code, but the correct code at the correct time. This will require a significant amount of forethought on the part of the designer. This is also why so many SMP-type multi-core solutions will be buggy or fail entirely.
There is no silver bullet here. No language extensions or class libraries will save us from ourselves. We, as designers, must take the time to understand the nature of SMP and learn the proper use of synchronization primitives and how to enforce sequencing of code on multiple cores. Fortunately, with the availability of Intel's Core 2 Duos, multi-core test hardware is relatively cheap. Unfortunately, there are no texts or classes that teach how to write good, multi-core code. For the time being, experience will be the primary teacher.
So, if you believe that you're going to be writing multi-core aware applications in your future, you've got to get on multi-core hardware and start testing your understanding of the environment. Look at processor/interrupt affinity issues and get up to speed on synchronization primitives such as message queues, binary/counting/mutual exclusion semaphores, and condition variables to understand how their use can change the interleaving of code.
Finally, pick up a core duo machine, install Linux and start playing with multi-threading code using the POSIX threading services. Also, plan on attending some of the sessions at the Embedded Systems Conference in April. There will be several sessions there that will help you understand the issues. A little time spent experimenting now will save you weeks of debugging later.
Palm Foleo is Dead
Well... From my last post I reported that Palm had announced the development environment for for the Palm Foleo was stripped version of Wind River's WorkBench product. I also stated that I thought that shipping the Foleo with a 2.4 kernel was a mistake given that 2.6 has been around for over 3 years now. In addition, I commented that the WorkBench environment cost as much as the Foleo itself, so I didn't know how that was going to fly in the open source community. Well, less than 2 hours after I made that post, Palm canceled the Foleo and made the claim that part of the rationale for canceling the Foleo was that the development environment was "too different" from their current and previous products.
Palm now claims that they are going to take a $10MM hit to write off the Foleo and begin working on the Foleo II. When I spoke to the Palm folks at LinuxWorld, I had suggested that they have an ultra-cheap development environment based on standard GNU tools to entice the hobbyist and traditional Linux developers to write/port code to the platform. Maybe that's what they'll come up with for this next product.
In spite of the issues with the Foleo, I believe that the time has come for a non-x86 based ultra-portable that runs Linux. A product like the Foleo (or new Foleo II) could be the impetus for non-x86 Linux to really start to take off and start on-par with the x86 flavor. Let's hope that even though this one is dead, that the next product, whether it's from Palm or someone else, is better thought out with the needs of the Linux development community in mind.
LinuxWorld and Mobile Devices
Greetings!
It's been about 6 weeks since my last posting. Sorry for the delay, but I've been at LinuxWorld and spent the past couple of weeks in Alaska with my family. Alaska was awesome. But, since I didn't strike it rich panning for gold, I'll have to keep working my day job :-).
The topic of this posting is LinuxWorld. This is the traditional gathering of all things Linux in San Francisco, CA. Typically, this show is comprised of people in penguin suits, mock battles between the distribution supporters (SUSE vs. Red Hat, etc.) and a lot of geek games. However, there was a very different feel to it this year. This year the show seemed to be dominated by the business types.
First, LinuxWorld was being held in conjunction with the Next Generation Data Center (NGDC) conference. The NGDC conference is all about how data centers are evolving and what technologies such as virtualization are going to do to the data center. I'll talk more about virtualization in an upcoming post. Suffice to say at this point that virtualization will be a factor in most of our lives at some point or other in the near future.
However, the other trend I spotted at LinuxWorld was the emphasis on the use of Linux in mobile applications. There were no less than three separate day-long presentations on the use of Linux in the mobile space. Motorola, Access and Wind River were all there talking about their strategies for Linux in mobile applications.
Motorola was handing out developer's kits for their new Linux-based phones like the Razor V8. I saw one of these phones running... I might retire my Treo to get one. It was quite slick.
Access (http://www.access-company.com) was showing their PalmOS-like layer for Linux. They are very specific about the fact that it's not PalmOS running on Linux ala emulation. But, most of us will not be able to tell the difference. And, our favorite PalmOS apps will run on their product! Now, to get a device that actually runs this...
Wind River Systems also had a day wherein they talked about their WorkBench product and their strategy for Linux in the mobile space. Of course, WR's WorkBench is based on Eclipse with many enhancements from earlier products such as SNIFF+. WorkBench is really a nice implementation for cross development. It runs in both Windoze and Linux and it has support for WR's JTAG solution. So, you can go from new board bring up to applications development in one environment. I just wish they could bring the per-seat price down to something affordable to small companies.
However, along those lines, WR is providing the development environment for the new Palm Foleo devices. These gizmos are either small, ultra-portable laptops, or they're really big Sidekicks depending on how you look at them. Based on the ARM processor, the Foleo has a pretty good keyboard and a decent screen and is targeted at those folks who don't need a full-blown laptop, but hate accessing the Internet on their phone. Unfortunately, Palm is running a 2.4 kernel on these things. I'm not sure why they chose 2.4 over 2.6 especially since 2.6 has been out for over 3 years now. But, the Palm folks assure me that 2.6 is in the works.
The WR WorkBench development environment for the Foleo has been significantly hobbled. Priced at around $500 for the environment, it will only allow you to develop applications code. No drivers or kernel code. And, the development environment costs as much as the Foleo itself. I'm now sure how this strategy will play with the Linux development community, but I guess that Palm will find out soon enough.
The other piece of the mobile market that I saw addressed at LinuxWorld was the issue of power management. Linux has long been crippled with respect to power management because of trying to fit into a Micro$oft-centric ACPI model without being able to gain access to the secret sauce of ACPI internals. However, significant progress is being made at supporting very aggressive power management techniques in Linux.
We now have the "tickless" kernel. No more periodic clock ticks that wake the processor up from a nice nap just to announce that the clock has advanced and there is nothing to do. With the tickless operation, a interrupt is scheduled to go off in the future that then corresponds to an event like communicating to the cell tower. When that event occurs, all of the other counters are updated.
Another advancement that was discussed at the conference was the new PowerTOP system. This tool was announced by Arjan van de Ven from Intel and is targeted at finding the top power consumers in a running Linux system. This is an important step forward as it now allows Linux developers to find some of the really stupid things we were doing and eliminate them. For example, the Gnome window manager polls for events at 100 Hz. The GVim editor also polls for events as does dozens of other applications even though the window manager was already polling for those same events. All things considered, there are thousands of applications that are duplicating work and causing the CPU core to wake up when its not needed. PowerTOP allows us to find these offenders and rewrite key sections to avoid this behavior.
The results from the PowerTOP efforts are already starting to bear fruit. Many of the most common applications have been modified and upstreamed from the distributions. This means that distributions such as Ubuntu "Gutsy" will have these enhancements when it comes out in October. With these modified packages, we're already seeing more than .5 hours of increased battery life on the test laptops. Now, if we could just break the ACPI stranglehold...
OK, so that's enough for this set of ramblings. Stay tuned for more discussion on virtualization in the embedded space as well as a more detailed discussion on power management.
Happy hacking...
Will GPLv3 make Linux irrelevant?
Well, GPLv3 has finally been released. This is an attempt by many of the movers and shakers in the open-source community to close the loop holes found in GPLv2. However, as someone who is trying to use Linux in many different types of embedded applications, I find GPLv3 somewhat troubling.
The reality of the embedded space is that silicon vendors such as Marvell, make it almost impossible to get data sheets for their parts -- requiring NDAs galore. And, vendors such as NVidia and ATI won't release their information at all. Whether or not you believe in the existence of intellectual property (IP), these manufacturers do. Hence, if you want cutting edge components in your devices, you have to play by their rules. I'm not happy about that, but that's the way it works in this business.
The reality of the embedded space is that we're trying to move hardware. Routers, cell phones, etc. are what we sell. The software that makes that hardware work is important if it cuts time to market. But, on the whole, the software/OS is a means to an end for most manufacturers. And, I feel that Linux is gaining popularity at least partly because we're seeing it in so many new devices that people interact with every day.
This is not to belittle the efforts in making desktop Linux a better product. I've been running Linux on my laptops for over 6 years now and it's better than ever. However, it's still not something I'd give my grandmother and expect her to be able to send emails with it. You still need a broadband connection and a lot of patience to work with Linux and all of the packages that need to be installed and configured. That's simply beyond most average consumers.
Nonetheless, I've got an NVidia GPU in this laptop and I want it to be able to run the 3D applications I want/need to do my work. Consequently, I use NVidia's proprietary driver and it works just fine. I also use Intel's IPW3945 wireless driver that also has proprietary hooks in it. GPLv3 would attempt to close the door to these proprietary drivers.
Now, the die-hard GPL folks would say that allowing for proprietary drivers is a slippery slope. If you allow proprietary drivers, then it's no longer open source and soon you'll have all kinds of patents that preclude the use of the software. The reality is that I just want my hardware to work. And, I'm not willing to wait for the 3-5 years that it would take for a "new" hardware vendor to come onto the scene that builds open hardware. Additionally, since Linux is such a small piece of their overall business model, I don't see companies like NVidia going out of business simply because we choose to use a vendor who open sources their driver -- if you could even find a laptop that used such a chip. But, I digress...
Embedded device manufacturers want software that enables them to move iron. The argument about royalties is pretty much OBE these days. Most O/S vendors have provisions for royalty-free code distribution. Instead, you want an O/S that enables you to move quickly, has developers available and can be tweaked to fit your application. Linux does this as long as you don't get draconian on the "no proprietary code" thing.
Fortunately, Linus and many other key Linux developers have not been too keen on GPLv3. But, there have been many attempts to limit proprietary drivers in the kernel by using the EXPORT_SYMBOL_GPL option on key APIs such as sysfs. This drives manufacturers to use their own power-management interfaces via /proc rather than potentially invalidate their intellectual property by using the existing power-management interfaces in /sys. This does not help the industry, nor does it help Linux, IMHO.
So, if Linux goes the way of GPLv3 or becomes too difficult to deal with WRT protecting your IP, manufacturers will simply switch to a different O/S. Unlike hardware manufacturers where there might be 3-4 vendors of a particular type of silicon, e.g., wireless interfaces, there are hundreds of different O/Ses out there. Linux is gaining momentum because vendors are using it. The community as a whole benefits from this shift.
If you make it too difficult to live within the bounds set forth by the licenses, manufacturers will move to something that is easier to live with. Therefore, Linux will become increasingly irrelevant in the embedded space. Another aspect that does come into play here is the thought that, "if my phone, MP3 player, GPS, etc. are all running Linux, maybe I should try Linux on my desktop as well". This is helping expand the use of Linux on the desktop. If Linux disappears from the devices we work with every day, then it disappears from our thoughts as well. And then, Linux runs the risk of withering on the vine.
Don't get me wrong here. I'm absolutely in favor of open source and the goals of the open source community as a whole. Given a closed source program vs. an open-source version of a program with equivalent capability (I'm even willing to take a little less capability), I'll go with the open-source program. But, I'm also a pragmatist. When it comes to hardware, silicon vendors control my ability to create interesting, desirable devices. I have to be able to work with them or I have no product.
Blind dogmatism to any position is rarely a good thing. I'm simply saying that the community should use a little common sense in determining how to move forward. Otherwise, we run the risk of making Linux essentially irrelevant in the embedded space, which has impact on it's relevance on the desktop. Think about all of the aspects of a decision such as licensing. We don't need another repeat of the Unix-wars of the 80s/90s. That rift opened the door to Micro$oft. I certainly don't want the Linux community to become famous snatching defeat from the jaws of victory by internal squabbles.
Carrier-Grade Linux in Space
Spacecraft applications have long been the bastion of proprietary solutions. Custom RAD-hard processors, custom bus architectures and custom command and control software are the typical fare of space-borne platforms. The use of all of this custom hardware and software makes access to space an incredibly expensive venture. All of this is before you can find a lift vehicle to get you to orbit!
There is no doubt that space is a difficult environment to try to field a system. Radiation, micro-meteors, space junk (debris), anti-satellite weapons and more can adversely effect satellite systems. In addition, it's difficult to send up a repair guy to take a look ;-). The general approach to spacecraft design so far has been to find a collection of hardware that works and then not mess with it. Not that this is necessarily a bad approach, but it certainly stifles technology development.
For example, the RAD 750 (a radiation hardened PPC 750) has been the "cadillac" of space-born embedded processors for years. This processor was released in 2001 and runs at a whopping 133-166 MHz. But, what's really amazing is a price tag that's measured in 100s of K dollars per unit. Alright, there's certainly been a good track record for this processor, and I can understand that nobody wants to rock the boat by switching to something untried in such a difficult environment.
However, we desperately need a way to increase the performance of satellite systems and simultaneously reduce the time to orbit for new designs. It's time the satellite community realize that maintaining the status quo is not helping accomplish the missions of Operationally Responsive Space (ORS). In ORS, we need to be able to pull a satellite together using COTS hardware/software and get a satellite to orbit in a matter of a few months rather than a few years as it is today. Carrier-grade Linux (CGL) can help us accomplish this.
The telecommunications carriers are pouring considerable resources into CGL and COTS platforms in an effort to ensure > 5-nines availability of embedded systems. What is 5-nines? No more than 1 hour of down-time for 100,000 hours of power-on time. Or, more practically, no more than 5 minutes of down-time per year. How can they do this? They use inexpensive processors (lots of them) and couple these processors with redundant hardware like power supplies, communications interconnects, and lots of already developed software for handling message services, fault isolation, cluster membership, etc.
ORS missions are typically short, low-earth orbit (LEO) affairs in which the satellite is expected to be on orbit for less than a couple of years. The use of redundant COTS hardware that could be left in the powered off state until needed could be leveraged to deal with the rigors of the space environment - and do so with less weight and power requirements than the current generation RAD-hard systems.
Some pockets of the space community are starting to come around. They're at least trying to understand how to use COTS hardware in space. But, the software tends to continue to be the dominating factor in the spacecraft development process. It's time to look at the COTS software as well and build upon the experience we've seen in the CGL telecom area and apply it to other areas.
What do you think about this? Any other ideas of how to apply existing technologies to space applications?
