Using parallel to multi thread scripts

Traditional awk, sed or grep commends do not multi thread by default. Multi Threading a task on a large data set can improve the time up to 50%.

For additional information this package: http://www.gnu.org/software/parallel/

Dependencies: 

Debian 

sudo  apt-get install parallel

CentOS 6

wget http://download.opensuse.org/repositories/home:/tange/CentOS_CentOS-6/home:tange.repo

yum install parallel

Some use cases:
You want to find a string in text:
cat ./logs.csv | parallel  awk ‘/string/’ > ./stringoutput.csv

This parallel contains no additional functionality.

Lets say that you want to add the ability to break the file up into block sizes. After adding –block and size of block (ex. 100M, 10M, etc) you must add –pipe

cat ./logs.csv | parallel –block –pipe 100M  awk ‘/string/’ > ./stringoutput.csv

Since parallel is focused on maximizing the threads you can limit this by using: –jobs

By default –jobs is the same as the number of CPU cores. Arguments such as:

cat ./logs.csv | parallel –jobs 1 –pipe 100M  awk ‘/string/’ > ./stringoutput.csv

This will limit job to single CPU thread.

Summary:

parallel is very powerful addition to scripts that require additional or focused resources.

Source

 

Using parallel to multi thread scripts

Installation of Fedora 23 on HP EliteBook 840 G1

Before I start with the brief tutorial I want to just list the configuration, results may vary based on the specs of the machine.

  • 14″ inch 1920 x 1080 display
  • i5-4300U 1.9 Base clock 2 core 4 thread
  • 8gb DDR3 12800 ADATA RAM
  • 180GB 530 2.5′ Intel SSD

I decided to use the latest version of Fedora based on the good experience that I have had with my desktop. The latest version of Fedora runs a very clean, stable and beautiful version of GNOME.

First, I downloaded the .iso and burned it to a USB drive. Since, the laptop might either ship with Windows 7 or 8.1 the BIOS might be set to either UEFI or Legacy Boot. In order to check this power off the machine, at BOOT when at the HP logo screen select ESC.

BIOS1

Once you have selected ESC. You will see a list of options, you will select the option f10.

IMG_0072BIOS3

This will take you to the BIOS setup screen, go to the Advanced Tab at the top right corner.

BIOS4

Make sure that USB boot is selected.

BIOS5

Scroll down further to the Boot Mode and select Legacy Boot. Once you have set these settings make sure to save and exit. You should now be able to boot from your USB bootable drive.

Most of the devices worked out of the box, the only devices that did not were:

  • Sound
  • Fingerprint Reader
  • f8 Mic Disable hot-key (There were so few I had to list it.)

Before I got into resolving the issues with these devices I wanted to get all the updates:

sudo dnf update

After all the updates finished I went to install Google Chrome, Google has really made this easier than in past years.

https://www.google.com/chrome/browser/desktop/

It should open in Software Updater, then select install and enter root password.

There are a few guides to “Things to Install after Installing Fedora 23”

It will vary based on your usage what will want to install, however here is a link:

http://www.binarytides.com/better-fedora-23/

The issue with the sound driver was related to an incomaptibilty issue with PulseAudio. I was easily able to resolve this by simeple Terminal Command:

sudo dnf|yum remove alsa-plugins-pulseaudio

Once I rebooted the issue was immediately resolved. I have tried so far without success in the past few hours since I have had the machine to probe the FIngeprint driver to see what drivers might exist for it. I will update this blog if I do find a fix for this. Please let me know if you think there are any other important additions.

Installation of Fedora 23 on HP EliteBook 840 G1