Embedded Linux can be used to craft impressive solutions, but it cannot just be taken, installed and used.
It has its own set of rules and there are certain factors that have to be considered before, and during, any installation. These factors include security, hardware, data, patches and application proficiency, and many solutions are incapable of delivering on all these essential fronts.
Linux is not a real-time operating system, even with the real-time extensions and real-time scheduler enabled. For anyone looking to build a real-time solution, this is not the one you are looking for. Interrupts are scheduled with the co-operative scheduler and so the user space is not interrupted for kernel tasks. In addition, context twitching only happens on time-slice boundaries. This means that interrupts will be synchronised with the kernel time-slice which makes for a worst case scenario of up to 10 milliseconds on embedded systems. The CPU has to be self-sufficient and capable of handling this type of latency in order for it to deliver the real-time capability required. This might mean that more DMA channels may be needed in order to compensate for interrupt latency.
When building solutions that will be moving a lot of data around on a regular basis, such as protocol converters, routers, data-loggers and mediating gateways, the I/O bandwidth is a concern. It will bring all of those cache pipelining, pre-fetching, branch predicting 4000 MIPS CPU amazingness to a grinding halt while it services the SPI data Flash or reads files from the SD card, waits for the DDR controller to load the next 2 MB of video to decode. I/O clock speed becomes very important as the system bandwidth will approach this, rather than the bandwidth of the CPU core. It’s worth looking out for devices that provide significant dedicated DMA or automated data transfer so as to deal with any latency and bandwidth issues.
The next concern is kernel support for the SoC and other peripherals – it is essential that any patches applied to the patched kernel provided by the vendor or any third-party supplier is incorporated into the mainstream kernel. Should there be a need to build a custom kernel this will only be possible if these changes have been pushed upstream in the kernel source tree. The SoC and other peripherals used in the kernel must be supported by the mainstream kernel. Kernel driver development, while interesting, is not a trivial undertaking.
Other considerations that need to be taken under advisement include the need to consult the kernel source and configuration files for supported device configuration. Some devices may only be supported on SPI bus and not on I²C etc. This has to be kept top of mind when developing hardware as it will make system integration a lot easier.
When it comes to the kernel and the tool chain, make sure to use an LTS kernel version. It is worth visiting Kernel.org which provides a list of long-term supported kernel versions. Whether making use of a kernel vendor, third- party or the vanilla Linux distro kernel, ensure that the kernel can be reproduced reliably within the development environment.
Linux can require plenty of non-volatile storage as well as RAM, and no matter what the manufacturing specifications on Flash devices are, JFFS and UBIFS are not viable options as random access root file Flash-based systems will slowly get corrupted and there is no way of repairing it in the field. Bitrot will kill your product.
At Keystone Electronics, we use JFFS only for rescue file systems, and in this instance also in conjunction with a RAM overlay file system in order to minimise write activity. For root file systems we usually use an ext4 formatted SD card. Although reliability of SD cards may be an issue, which can be solved through careful procurement, the availability of user space file system tools for the root file system makes life in the field a whole lot easier.
When it comes to embedded Linux distributions there are a number of challenges that could impact an installation. It is essential that the system used is one that is reliable, tested and capable, such as Blue Penguin. The latter is a robust and ubiquitous solution that can be maintained, developed, customised and is eminently sustainable.
An out-of-the-box embedded Linux solution means that there is no need to roll one from scratch, wasting time and energy on something already done. However, there are some factors worth keeping in mind before investing. Find ways of reproducing at least some of the packages in the distribution so the option to rebuild and repackage applications completely, even against their given dependencies, is possible. Also ensure that the distro is capable of upgrading applications and images in the field; an inability to perform this could be disastrous.
Within the development environment and custom application development there are some considerations that need to be taken into account on the distro. It is not always necessary to develop native C or C++ apps as it is possible to get away with scripting languages so make sure that the distro has proper scripting support for languages such as Python and C#. When developing native C++ applications using cross compile environments for the distribution, keep in mind that there is a need to install or build the run-time dependencies of the application into the cross compile tool chain.
The embedded Linux distro needs to have a way of distributing kernel modules and providing a framework to deploy them. This is necessary if, at some point, there is a need to extend kernel modules with extra functionality without having to completely rebuild, as most times this is not possible due to a lack of space on Flash to do so.
Another consideration is application distribution as there has to be a way of distributing applications, either through a native package manager or through an existing framework or infrastructure. The latter is not a recommended course of action. In addition, the distro must have sustainable field support as at some point there will be a need to roll out upgrades and fixes. The distro must be able to handle this, such as Blue Penguin’s capability in doing so via snapshot releases.
It is worth bearing in mind that when upgrading in the field you are dealing with a complete ecosystem of applications, libraries and protocols that must work in harmony. It is not just one application that requires support so it is essential that this synchronicity is conserved. It is necessary to have mechanisms in place in the event of catastrophic storage failure as it can be hard to recover – you don’t want to have to take things out of the field to repair them as it is preferable to go into some sort of ‘save rescue’ mode through which the system can be recovered.
Finally, when looking to invest in an embedded Linux solution the issue of security cannot be overlooked. A reliable system will ensure that security is a priority across connectivity, the protection of data and the development of systems that prevent hacking. With Keystone, our Blue Penguin ticks all the boxes of a robust distro and, to really round things off, we also provide an array of support and services for Guinnux, that forms part of the embedded Linux solution.