CSV file breakdown

In a recent project I had a massive CSV file, and by massive I mean 2.8 million rows long. Unfortunately I needed this file broken down in to multiple files to process, so I looked to the masterful Google for help. Here is how I found to breakdown the files on a Mac and Microsoft machines.

Mac File Breakdown

To do the breakdown on the Mac machine it is super easy. If you put the large CSV file in a folder and then navigate to it in Terminal. Then in a simple move you can drop in the following command into Terminal, with replacing the {LinesCount} and {FileName} with their respective content.

split -l {LinesCount} {FileName}.csv

Windows File Breakdown

On the Microsoft side it is a little harder, but I found this good source to do the job. With this little trick, you do the same as the above. Put the CSV and a new bat file in the same location, replace the {LinesCount} and {FileName}, then run the bat file with the code below.

@echo off
setLocal EnableDelayedExpansion

set lineLimit={LinesCount}
set fileSrc= {FileName}.csv

set lineCounter=1
set filenameCounter=1

set name=
set extension=
for %%a in (%file%) do (
set "name=%%~na"
set "extension=%%~xa"
)

for /f "tokens=*" %%a in (%fileSrc%) do (
set splitFile=!name!-part!filenameCounter!!extension!
if !lineCounter! gtr !lineLimit! (
set /a filenameCounter=!filenameCounter! + 1
set lineCounter=1
echo Created !splitFile!.
)
echo %%a>> !splitFile!
set /a lineCounter=!lineCounter! + 1
)

Accessible Table Generator

Content Editors entering content is the bane of all developers lives, as the Editors will add in the most ill-systematic mark-up through the Content Management System (CMS). These make the pages look wrong sometimes, they put unordered lists in paragraphs, they put as many top header in as they can to stress it’s importance and many more annoyances. However they are not developer and do not know what they are doing, that’s why I build an Accessible Table Generator for the Editors at my work.

Tables are a big issue with accessibility because of how they are used. Back in the day this element was the god father of elements, as it was used everywhere to structure content. It was the best and the most simple method to make sure everything stayed in a format required for displaying on webpages and even e-shots. In most cases you could breakdown a website to fit a grid system, especially in older websites, so the only way to make this grid without using a lot of CSS and spans was the ‘table’ element. However times have moved on and standards have improved with the technology. Now developer are using the ‘division’ element (div) to structure pages. They have a lot of flexibility, there is less default styling to them, it take less elements to replicate structures and they are easier to work with, compared to the ‘table’ element. That’s why us developers use them instead, but Content Editors do not know of these or how to use these for their content. It is not their job to know this and that is why when they want to structure anything they use the WYSIWYG table generator. This then for them, puts the content in the structure they want and its easy as hell.

This however is not always the best method for what they truly need, not that the Editor would know. Straight off a table should not be used to structure content because of the difficulty them. Tables are hard to style as they have a lot of default CSS that needs to be overridden before you can even start on your own styles. With the hard styling and the standards compliant way you use the tables, it can be hard to format the contents in the way that it is required, as example below:

running

This may look like a great idea to have in a table format for the layout, as you have a top cell full width with content in then two other cells. One with an image and another with some more content. However if you look closely at this then a table would be very hard to use because not only first would you need to fight the built in styles, but the bottom content you would put in cells are not evenly sized. Trying to make some of the cells 50% of the full width and some 20/80 split is impossible without nesting more tables within more tables. As well as this, if you was to change the content and add more in, it would not flow. As you add content to this idea above, you would need to decide which block the content goes in to. These are just some of the problems you would come across with then simplest of design, but with the more systematic mark-up approach you can fullfill this with less HTML and some CSS. As you can see from the code below, I have got a container for all the content, so if it was to grow then it would retain the structure and styling. The image tag is place in where I want the content to flow round and then with CSS is floated left.

Ut lacus orci, aliquam a lacus sit amet, aliquet pulvinar odio. Integer quis ex et nulla vestibulum consecte

quis et quam. Donec eleifend felis non ultrices pharetra. Vivamus non dolor aliquet, suscipit nisi si amet, vehicula nisl.

People Running

Ut lacus orci, aliquam a lacus sit amet, aliquet pulvinar odio. Integer quis ex et nulla vestibulum consectetur quis et quam. Donec eleifend felis non ultrices pharetra. Vivamus non dolor aliquet, suscipit nisi sit amet, vehicula nisl. In nec pretium mi. Aenean sit amet arcu maximus, gravida orci non, dictum odio. Curabitur a nunc ipsum. Nunc sit amet leo urna. Donec in sollicitudin mi. Proin ac lectus orci. Sed scelerisque gravida est, at posuere orci malesuada vel.

Not only is it harder on the developer to use tables, but it is not what they are intended for. The World Wide Web Consortium (W3C) say in the Web Accessibility Initiative (WAI) that ‘tables are used to organize data with a logical relationship in grids’ (https://www.w3.org/WAI/tutorials/tables/). This means you should only use the tables when the content is made to be in a grid format and not only that but for the ‘table’ element it should only be tabular data. For non-tabular data you can replicate the table with division elements and CSS to structure it like a table.

When to use ‘tables’ or ‘divs’ then?

Use the table element when the content that will be entered is tabular. This can be content that you would normally put in a spreadsheet or a record of data.

Use the ‘div’ element when the content needs to be placed in a strict grid format, but is not tabular data. There is some discuss if this is ever required as you can always use CSS to format the content in that fashion.

Some people decide over the ‘table’ element for all data due to the difficulties with the element compared to the ‘div’, and also to the responsive issue on mobile devices. However there are plenty ways you can work with tables to make them easy and responsive. One way is to fully override the tables default instinct at a media size point and make them perform like division’s with the proper use of a table, as shown by CSS-Tricks.

The Accessibility Table Generator!!! This nifty little tool that I have built generates the table or division structure you need with all the accessibility required. Use the User Interface to mark out the table size or use the form fields. You can then decide on what kind table you need for your data. It could be one where the top row is your heading data, where the first column is your heading data or where both of them are. Next you decide type of mark-up you need and finally add a caption for the table. The caption will describe the data in the table.

tg1

Once you click the ‘Generate Table’ button below a grid of text area’s will appear. These represent the table with the grey fields being the heading fields in the table. You can now enter all the content you are going to add in the table before it generates the code.

tg2

When you are happy you can click the ‘Generate Accessible Table Mark-up’ button, which will generate the desired mark-up for you to copy and use.

tg3

Here are some resources that also could be useful:

Less code and more tools

In the world of automation, DevOps and a huge amount of tools for website development are we doing less code and so becoming less of a developer and more of an admin?

Over the few years of me doing web development I how found more things have come out the box to reduce the amount of coding required. There are good reasons for this as it has sped up dev to live timelines as we can push things out faster and also us as developers don’t need to do repetitive changes, like navigation or text, as they can be done by content editors.

However it does then lead me to worry about what the future holds for us web developers as it means we will be needed less or only for the small changes, unless we are the ones building the tools. I have moved from a company that has an in house built Content Management System, which we would have to build, support and update all the time. As new clients came in and new advancements were made in other CMS that meant we had a challenge to get caught up. This also meant we were constantly busy and always looking for future improvements. I do like this work pace as I never like to be bored or without work to do, but it does then mean a lot of time spent. There can be a whole host of bugs and issues that constantly stream in, especially if you are working in a digital agency because it’s not just one customer finding bugs. This can then stop the growth of the CMS and the company, so we all turn to the companies that can provide the basic CMS that we can the advance and develop to what we need.

These can be WordPress for the PHP developer or even Kentico for the .Net developer, both really good at what they do. They, and a bucket load more, provide a very nice CMS that can easily be managed by the content providers. If there is a bug with their code then they will fix it and give out the updates. If they develop a new tool to make things better, then it will be given out to everyone as an update. This means there is less work by us developers for the basic little things and more time for the improvements that we want. We can be making new modules, style changes or just about anything we want without worry that something is going to fall over. The trip up though is how much new things does a company need? When the CMS company provides the basic needs of a content editor then there is not much left  for them to need, especially every day to fill out your time. You think about any website and how much they actually change, not much in development more just content.

This to me starts to point to how we are going to need less developers, but more smart developers. It’s the same old saying of Quality Over Quantity. Before every company needed developers to keep up with the tide of work and development needed for the website, plus services building. They were the hot new thing that every company needed to be a proper company. Now though with more companies turning to pre-build products and social media for their content outlet, there is less need. The only need will be for the smart developers to be building updates to the products the companies are using, but how many developers does Squarespace or WordPress need.

Luckily I think there will always be demand for developers, either with web or transition to other sectors. What do you think?

 

 

 

Can Project ARA work?

Just about all developers watched and followed the Google I/O 2016. You heard about the Allo, the Duo and the Google Home air freshener, but something they also squeezed in was a nod to Project ARA. I thought this was dead with no recent news, but they have brought life to the project again so I ask, can this idea work?

For those who don’t know about Project ARA, it is an idea from Google to build a modular phone. Their plan is to have a phone that you can create and be your own. All you will have to do is replace, upgrade and buy new modules for the phone to keep it running. If you want the new camera upgrade, then you don’t need to buy a whole new phone. You just need to pop to a store that sells the modules and then slot it in.

They also say on the Project ARA website, they are looking to expand to the Open Marketplace. This could mean they will expand to allow third parties to build for the ARA phone. Google said at the I/O that they plan for the phone to be on sale in 2017, so we don’t have long to wait. The question about the open marketplace would be if they will have enough companies building for it before it comes out. If the phone drops and there is nothing other than Google things build for it, then it will be a hard sell and an uphill battle to reach the general public.

 

Why Could It Succeed

Google are known for throwing ideas at the wall and seeing what sticks, so what makes this another egg at the wall and not the new best thing of 2017? I think a lot of it has to do with cost, flexibility and personable. The whole idea is to be able to swap out your modules whenever you want, which brings in the flexibility and the personalisation. Most phone contracts for the general public are 2 years, so when we see a new phone in the middle of this, all we can do is dribble. If they announce a new feature that can be added by a module, then you can get that upgrade then. This also means updates can come faster that rely on hardware. For example, if Apples iOS Force Touch just needs the newer hardware to run that feature, with this you could get told what modules you need to update to get that feature, then do it.

Speaking about another company, as said before they want to open the development of modules to other companies. This could be great to see what they build for the phone and what could be useful. Like the big hype is at the moment, you could have better fitness tracking or blood sugar reader for people with diabetes. The realms are limitless, only bound by how many custom modules you can fit on the phone. It would depend how much room the basic modules you need take up to then how many custom modules you want can get on the phone. The benefit of these being small, easy fit modules though, is they can all be in your pocket for an easy swap over. The phone would just be limited by the imagination of the companies making the modules and by the cost of them.

We would hope the modules don’t cost that much as you would probably want to buy a few at a time and get new ones frequently like apps. If these modules are expensive then people might think why not wait for the next best phone, which will have all the upgrades at once.

Buying the phone will be interesting as well, because you are going to be the creator of the phone. It would be cool if you can basically spec out your phones modules and then put a price to it. For example £200 for the basic phone then 5 X £50 for each module. If you wanted a cheaper phone, you could then either downgrade the modules or you could just get less modules. The flexibility of these devices and the pricing would bring smart phones to more people, while also spreading the rate of updating as well, if they do it right. I think these factors could make the project ARA a starting point for other manufactures to follow.

 

Why Would It Fail

Project ARA is so flexibly it is amazing, but do we really want it that flexibly. You and me will probably say hell yes, but a general public user which is the main stream manufacturers are trying to hit might not. Think about Microsoft App store, no one wants to build apps for them as no one really wants their phones. They have even had to Open Source and partner with companies like Xamarin to make it easier for developer to build for them.

Does the general user know about Camera or Speaker quality and specifications? No and they won’t want to learn about these technical things. You may also find that the sales people don’t know either. In university I applied to a phone shop sales job, told them I knew loads about the handsets and their OS, but was told you don’t really need to know them things, you just need to sell the phones. So how will they sell a phone and modules that they don’t even understand? I think it will be a hard sell to an average user unless it can be sold as a package. They just want a cool, fast phone with awesome gadgets.

Depending on that would then be interesting if the companies making the modules, or Google want to make modules, actually want to. If they can’t see potential of the average user buying their module, then why would they spend time and resource on the product?

 

Until It Begins

I think we will really have to wait and see how Google approaches this. If they can package it and sell it well to the general public, then the Open Market will be willing to put time and money to the project. If that all goes well then I see no reason why this could fail as it has what all people want, a flexible, personal and affordable phone. Either way I could see me getting this phone depending on the modules built and price.

 

Tell me what you think about Project ARA and how it will do when released?

 

Galen Framework Hover in JavaScript

A bit of a small audience for this one, but a project I was put on wanted us to use the Galen Framework to test their UI. For people who do not know what this framework is, it is a method of writing acceptance code that matches up to the output in the browser window. This can be coded to work for all different device, sizes and URLs. One hard space I came to was to action the hover state of an element and then test it.

If you would like to read up more about the Galen Framework first then you can find all the information you need on the website.

To test you can either run it in the Galen syntax to run in the Command Prompt or you can also run it using JavaScript. I opted of the Galen Syntax for no real reason, but there was a method to action the hover in the JavaScript version.

Ivan Shubin, the creator of the Galen Framework, point me here for the example.

load("init.js");

load("pages/WelcomePage.js");

 

testOnAllDevices("Welcome page", "/", function (driver, device) {

new WelcomePage(driver).waitForIt();

checkLayout(driver, "specs/welcomePage.gspec", device.tags);

});

 

testOnDevice($galen.devices.desktop, "Menu Highlight", "/", function (driver, device) {

var welcomePage = new WelcomePage(driver).waitForIt();

logged("Checking color for menu item", function () {

checkLayout(driver, "specs/menuHighlight.gspec", ["usual"]);

})

 

logged("Checking color for highlighted menu item", function () {

welcomePage.hoverFirstMenuItem();

checkLayout(driver, "specs/menuHighlight.gspec", ["hovered"]);

});

});

You can see the ‘hoverFirstMenuItem’ that actions the hover. This is built into the Framework that comes with the JavaScript. You can find more about the API and the ‘hover’ method on the Reference Guide for JavaScript.

However I was trying to use the Galen Framework in the command prompt, which can inject JavaScript into the page that runs once the page has loaded. The first idea I had then was to run a ‘hover’ action, which because the site I was running it on has Jquery loaded I could use like below:

$(document).ready(function () {

//hover links

$('a').trigger('mouseenter');

$('a').trigger('mouseover');

$('a').trigger('hover');

$('a').hover();

});

Though as you can see I went a bit overboard trying these all out as none of them seemed to work. I knew the Jquery was working as I was able to run other method to do click events and other thing. After some searching I found a method.

I found a way to get all the style sheets and then search through them. Once I could search them I could find out what the ‘hovered’ styling was for that element. Then I could extract the styling and inject it onto the element, so now its standard style becomes its hovered style.

You can see the below method that I use ‘document.styleSheet’ to loop through each style sheet, then get each rule and loop them. If the class I am looking for is found the I user ‘cssText’ to get all the styling from the rule. This does come back as ‘{background-color:blue}’ so the response text is stripped down to just the CSS.

function getStyle(className) {

            var c = '';

            for (var i = 0; i < document.styleSheets.length; i++) {

try{

                var classes = document.styleSheets[i].rules || document.styleSheets[i].cssRules;

                if (classes) {

                    for (var x = 0; x < classes.length; x++) {

                        try {

                            if (classes[x].selectorText.indexOf(className) > -1) {

                                var s = (classes[x].cssText) ? classes[x].cssText : classes[x].style.cssText;

                                s = s.substring(s.indexOf("{") + 1);

                                c += s.substring(0, s.indexOf("}"));

                            }

                        } catch (ex) {

                        }

                    }

                }

                        } catch (ex) {

                        }

            }

            return c;

        }

You will notice there is a few try catches in the loops, these are because some style sheets it picks up are not accessible so they create an error in the JavaScript. The try catches mean it can keep looping on through the ones it can access instead of breaking out. Another bug fix is where it uses ‘rules’ or ‘cssRules’ and ‘.cssText’ or ‘.style.cssText’. This is due to different browser standards, some of the browsers use one set and some use the other, this prepares for either case when your tests run.

Below is how the full code sits together and can be used. You can see I invoke the method by passing ‘a:hover’ as it is the hover state I need to access to the link.

$(document).ready(function () {

function getStyle(className) {

            var c = '';

            for (var i = 0; i < document.styleSheets.length; i++) {

try{

                var classes = document.styleSheets[i].rules || document.styleSheets[i].cssRules;

                if (classes) {

                    for (var x = 0; x < classes.length; x++) {

                        try {

                            if (classes[x].selectorText.indexOf(className) > -1) {

                                var s = (classes[x].cssText) ? classes[x].cssText : classes[x].style.cssText;

                                s = s.substring(s.indexOf("{") + 1);

                                c += s.substring(0, s.indexOf("}"));

                            }

                        } catch (ex) {

                        }

                    }

                }

                        } catch (ex) {

                        }

            }

            return c;

        }

        //all links

        $('a').attr('style', getStyle('a:hover'));

});