Sonntag, 25. September 2016

node.js: create and scale a simple rest microservice cluster with cluster.js and express to your servers cpu

Everyone speaks about microservices. Today we want to create our first node.js microservice based on an article in the current phpmagazin.

On top we want to add this microservice to a servicecluster, which can scale perfect to your hardware and doubles your number of request per second to 10000 hits / sec

Prerequisits

We need some things installed on the local machine.

node.js and npm installed

If you have installed the node and npm we can proceed

Create a npm project

-npm init

This will open a promt to fill your projectdata

Install needed node libraries

We need express.js as a webserver framework, SQLite.js as the database-layer and body-parser to process responses

-npm install --save express body-parser cluster

Create index file

Our index.js references the libraries, implements the router and starts the app on port 8085

Here is the index.js

const express = require('express');
const bodyParser = require('body-parser');

const app = express();

app.use(bodyParser.urlencoded({extened:false}));

require('./lib/router')(app);

app.listen(8085);

Create your routings

Our service implements 6 routes:

URL	HTTP Method	Description
/timetrack/	GET	READ all entries
/timetrack/user/:id	GET	READ a user entry
/timetrack/id/:id	GET	READ an entry
/timetrack/	POST	add an entry
/timetrack/:id	PUT	update an entry
/timetrack/:id	DELETE	delete an entry

Lets create this routes in the folder "lib" (create this folder) by adding a file "routes.js"

This simple router just takes the requests and returns a static string, so that we can see the router working

lib/router.js

module.exports = function (app)
{
    app.get('/timetrack/id/:id', (req, res)=>
    {
        res.send('Returning a specific item');
    });

    app.get('/timetrack', (req, res)=>
    {
        res.send('Returning all items');
    });

    app.post('/timetrack', (req, res)=>
    {
        res.send('add an item');
    });

    app.put('/timetrack/:id', (req, res)=>
    {
        res.send('updating an item');
    });

    app.delete('/timetrack/:id', (req, res)=>
    {
        res.send('delete a specific item');
    });
};

Create your cluster

Cluster.js gives us the possibility to create 1 nodeprocess per cpu. So you can scale your node app with factor 8 on a octacore i7

This will double the number of request per second.

cluster.js

var cluster = require('cluster');
var numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
  console.log('#######################################');
  console.log('found:' + numCPUs + ' cpus on this server:');
  console.log('#######################################');

  for (var i = 0; i < numCPUs; i++) {
    
    console.log('starting cluster instance on cpu:' + i);
    cluster.fork();
  }

  cluster.on('exit', function(worker, code, signal) {
    console.log('worker ' + worker.process.pid + ' died');
  });
} else {

    //change this line to Your Node.js app entry point.
    require("./index.js");
}

Test your service

node index.js

Open a second terminal and run

ab -n 10000 -c 1000 http://localhost:8085

Now we want to scale our service:
Stop your app and restart it with

node cluster.js

Restest the service and compare the results

ab -n 10000 -c 1000 http://localhost:8085

As you can see, in the first test we got 5448 Request per second. On the second test with cluster.js we have 10756 Requste per second. This is more then double the perfomance.

Realy awsome

http2: make express talk https with spd2

In on of our last tutorials, we implemented http2 in a node app with spdy.
Today we want to teach express how to speak http2. Except the cert creation it is a quite easy task.

To make things simple we selfsign this certificate and import it to firefox.

Prerequisits

We need some things installed on the local machine.

a /etc/hostfile entry for "nodejs.local"
a ssl certificate for nodejs.local
node.js installed

Get the ssl cert and the server key

Create a projectfolder in your webroot.

Save that cert as server.crt and the server.key to your local project under <projectroot>/etc/certs/

server.crt

server.key

Alternativly you can sign your own cert with openssl

Now open in Firefox:
Settings->Advanced->Certificate and import it under "Cert Authority"

Create an node-express / spdy app

Install spdy and fs

npm install --save fs spdy exppress

Spdy is needed to handly http2 requests and fs will handle our key and certfiles.
Express will do the routing and takes our requests. After that the express app is passed into spdy as an agument. On that point spdy will handle the express app and bind a server on port 8084

Create the webserver by saving folling code as index.js in your projectroot

const spdy = require('spdy');
const fs = require('fs');

const express = require('express');
const app = express();

const options = 
{
     key:fs.readFileSync('../etc/certs/server.key'),
    cert: fs.readFileSync('../etc/certs/server.crt')
}

app.get('/',(req,res)=>
{
    res.send('Hello express');
});

spdy.createServer(options, app).listen(8084);

Run your new http2 server

node index.js

This will bind a simple server on port 8084, so that you can surf http://nodejs.local:8084 and see "Hello Client"

If you open firebug, you will see, that your request will use http 2 on express

This is AWESOME

Samstag, 24. September 2016

ivy: publish to nexus - unauthorized

It tooked me 6 hours and many retries to find out, that its not possible to use the ivy publish task on a host like: "http://localhost:8082"

It shows up in a message like that: impossible to publish artifacts for com-test#coolsoftwaremodule;working@pboethig: java.io.IOException: Access to URL http://localhost:8082/content/repositories/releases/com-test/coolsoftwaremodule/1.2.3.4/coolsoftwaremodule-1.2.3.4.zip was refused by the server: Unauthorized
I wrongly have used a build.properties like that:

repo.protocoll=http
repo.host=localhost:8082
repo.realm=Sonatype Nexus Repository Manager
repo.username=admin
repo.password=admin123

and a credentialsobject like that:

<credentials 
     host="${repo.host}"
     realm="${repo.realm}" 
     username="${repo.username}" 
     passwd="${repo.password}"

If you want to ivy:publish to a custom port you have to configure the port like that:

repo.protocoll=http
repo.host=localhost
repo.port=8082
repo.realm=Sonatype Nexus Repository Manager
repo.username=admin
repo.password=admin123

and a credentialsobject like that:

<credentials 
     host="${repo.host}"
     port="${repo.port}"
     realm="${repo.realm}" 
     username="${repo.username}" 
     passwd="${repo.password}"

Hope that will save you some time!

ant: retrieve , resolve and publish dependencies to nexus

For everyone who is looking for the basics in Ant, Ivy and Nexus, here is my simplest example how to connect, retrieve and publish artifacts from nexus with ant and ivy

Prerequisits

We need some things installed on the local machine.

ant installed

apt-get install ant

nexus installed

https://github.com/sonatype/docker-nexus

Filestructure:

<projectroot>
    |__build.xml
    |__ivy.xml
    |__ivysettings.xml

Setup the build.xml

We want to cleanup the local build folders, download some needed dependencies and cleanup the build cache. A very simple task, you would think. But have a look. That will come a little bit more complex now.

All single tasks will be defined in the build.xml. This file will be run by ant.

Its important, that you see the "depends" attribute on every single task, This attributes defines the order of processing the single task.

My build.xml

<project name="Testproject" default="init" basedir="." xmlns:ivy="antlib:org.apache.ivy.ant">
  <description>
    simple example build file
  </description>
  <!-- set global properties for this build -->
    <property name="src" location="src"/>
  <property name="build" location="build"/>
  <property name="dist" location="dist"/>
  <property name="ivy-version" value="2.2.0"/>
  <property name="ivy.url" value="http://central.maven.org/maven2/org/apache/ivy/ivy/${ivy.version}/ivy-${ivy.version}.jar"/>


<target name="init" depends="clean">
    <ivy:settings file="ivysettings.xml"/>
</target> 

<target name="resolve">
  <echo>Downloading dependencies defined in ivy.xml</echo>
    <get src="${ivy.url}" dest="${basedir}/.ant/lib/ivy-${ivy.version}.jar" skipexisting="true"/>

    <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${basedir}/.ant/lib/ivy-${ivy.version}.jar"/>
    
    <ivy:retrieve pattern="target/modules/[artifact].[ext]" symlink="true" type="zip"/>
    
    <unzip dest=".">
      <fileset dir="target/modules"/>
    </unzip>
    
    <mkdir dir="target/buildlibs"/>
    
    <ivy:retrieve pattern="target/buildlibs/[artifact].[ext]" symlink="true" type="jar"/>
</target>

<target name="clean" description="Cleanup build directory">
    
</target>

<target name="clean-all" depends="init,clean" description="Clean and purge caches">
    <!-- Purge the ivy cache -->
    <ivy:cleancache/>
</target>
</project>

This sets up a "Testproject" in ant which uses ivy for configuring the build task properties. As you can see, we are defining some properties (src, build and dist).

There are 4 Task, which will be executed in following order.

clean
- this task cleans up the build folders. So all sources, which are only needed for this build can be deleted. Its getting called by the "init" task.

init:
- this is the task, which initializes the projectbuild starts the cleaning task first. As you can see, there is a file reference to ivysettings.xml. Later I will explain that.

resolve:
- this task will download all needed buildtools and project dependencies, we have defined in a the central dependency managementfile named "ivy.xml".

clean-all
- At the end we will delete the ivy cache.

To make that buildfile work, we need 3 further more files. A "ivy.xml", a "ivysettings.xml" and a "build.properties file".

Setup the ivy.xml

In our build.xml in task init we defined that we want to use ivy and it's settings as our configuration tool to configure the build.xml

If you have a look in the ivy.xml you will see following code

ivy.xml

<ivy-module version="2.0">
    <info organisation="com.test" module="test.php.project"/>
    <publications>
        <artifact type="zip" ext="zip"/>
        <artifact type="pom" ext="pom"/>
    </publications>
    <dependencies>
     <dependency org="com-test" name="test" rev="1.1.1" transitive="false"/>
    </dependencies>
</ivy-module>

As you can see, we have 3 object here:

the info object
- This will define the organisation, which can be the reverted company homepage like "com.google" and the modulename, which defines the name of the softwarepackage

the publications object
-this defines the artifacts , which will be published at the end of the process. Because we will use the maven2 standard we will publish a .pom file, which contains the needed maven2 metadata for our buildartifact. You will see, thet nexus needs that pomfile to store the packageinformation.

the dependencies list
- here you will define your project or builddependencies, which will be downloaded and usedduring the build. All 3rd party libraries can be defined here and merged into your project later by a pecific ant task.
In that project we will simply download a test.zip package from organisation "com-test" in version 1.1.1
Please make sure, you have uploaded a similar file to your nexus repo, so that we can resolve this dependency

Setup the ivysettings.xml

This file you will need to store common buildinformation like credentials to access the artifacttool (nexus) or to define your resolvers to to download and publish your artifacts.

This file gets included in your init task of your build.xml. Thats very important. otherwhise, your build task wont find its properties to work.

ivysettings.xml

<?xml version="1.0" encoding="UTF-8"?>
<ivysettings>
 <properties file="${user.home}/.ant/config/build.properties"/>

    <credentials 
     host="${repo.host}"
     port="${repo.port}" 
     realm="${repo.realm}" 
     username="${repo.username}" 
     passwd="${repo.password}"/>
    
 <settings defaultResolver="public"/>
 <resolvers>
  <ibiblio name="public" m2compatible="true" useMavenMetadata="false" root="${repo.protocoll}://${repo.host}:${repo.port}/content/repositories/php"/>
   
   <url name="publish" m2compatible="true">
      <artifact 
   pattern="${repo.protocoll}://${publish.host}:${repo.port}/content/repositories/releases/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"/>
  </url>
 </resolvers>
 <modules/>
</ivysettings>

As you can see, we are defining a property named "properties file". This means, that we will load our predefied set of properties from a file named "build.properties". Here we will store our secret accessdata to nexus. Every programmer has to install this file manually in his home dir

Further more you see a creentials object. This object uses the variables defined in the build.properties to store the accessdata to nexus. At last the ivysettings defined the url resolver. The resolver downloads and uploads the build artifacts to nexus under a defined url pattern,

Setup the build.properties

build.properties
repo.protocoll=http
repo.host=localhost
repo.port=8082
repo.realm=Sonatype Nexus Repository Manager
repo.username=admin
repo.password=admin123

Save this file to /<homedir>/.ant/config/build.properties

Make sure your nexus connectiondata are working!

Run your build

Now you can run ant init on the terminal

ant init
If everything went well you will get a Build success

Dienstag, 20. September 2016

ant: create targets for different os

If you want to execute your ant task on different os like windows or linux you can use a special exec attribute "osfamily"

This attribute allows you to define targets for each os.

Here is a sample target to execute phpunit on linux and windows.

server.crt

<project name="Testproject" default="dist" basedir=".">
  <description>
    simple example build file
  </description>
  <!-- set global properties for this build -->
  <property name="src" location="src"/>
  <property name="build" location="build"/>
  <property name="dist" location="dist"/>

  <target name="init">
    <antcall target="PHPUnit" />
  </target>


  <target  name="PHPUnit" description="Run PHP Unit">
    <exec osfamily="unix" executable="${basedir}/vendor/bin/phpunit" failonerror="true">
      <arg value="--configuration"/>
      <arg value="${basedir}/phpunit.xml"/>
    </exec>
      <exec osfamily="windows" executable="${basedir}/vendor/bin/phpunit.bat" failonerror="true">
          <arg value="--configuration"/>
          <arg value="${basedir}/phpunit.xml"/>
      </exec>
  </target>

  <target name="dist">
  </target> 
  <target name="clean"
        description="clean up">
    <!-- Delete the ${build} and ${dist} directory trees -->
    <delete dir="${build}"/>
    <delete dir="${dist}"/>
  </target>
</project>

Simply run:

ant init

As you can see, there are 2 exec objects with a osfamily attribute for windows and linux.

This is simply awsome !

Sonntag, 18. September 2016

https: use node.js spdy to implement http2

Because http2 is the next generation protocoll in the web with all its advantages like performance, better informations etc, we want to implement a simple http2 webserver.

Most of the browser do actually support http2. FF, Safari IE 11 and Opera in the latest version does.

Because all browsers are making ssl / tsl to the encryption standard when using http2, you will need a certificate.

To make things simple we selfsign this certificate and import it to firefox.

Prerequisits

We need some things installed on the local machine.

a /etc/hostfile entry for "nodejs.local"
a ssl certificate for nodejs.local
node.js installed

Get the ssl cert and the server key

Create a projectfolder in your webroot.

Save that cert as server.crt and the server.key to your local project under <projectroot>/etc/certs/

server.crt

server.key

Alternativly you can sign your own cert with openssl

Now open in Firefox:
Settings->Advanced->Certificate and import it under "Cert Authority"

Create a simple webapp using http2

Install spdy and fs

npm install --save fs spdy

Spdy is needed to handly http2 requests and fs will handle our key and certfiles

Create the webserver by saving folling code as index.js in your projectroot

const spdy = require('spdy');
const fs = require('fs');


const options =
{
    key:fs.readFileSync('./etc/certs/server.key'),
    cert: fs.readFileSync('./etc/certs/server.crt')
}

spdy.createServer(options, (req, res)=>
{
    res.writeHead(200);
    res.end('Hello Client');

}).listen(8084);

Run your new http2 server

node index.js

This will bind a simple server on port 8084, so that you can surf http://nodejs.local:8084 and see "Hello Client"

If you open firebug, you will see, that your request will use http 2

This is AWESOME

openssl: create a self signed certificate for a local host entry

From time to time you will need to create a TSL / SSL certificate on your local machine. For instance, if you want you use http2. In most of the browsers http2 implements TSL / SSL. So you need a cert for your localhost.

You will learn in this tutorial

to create a secure and a insecure serverkey needed for a Certificate Request with openssl
to create a Certificate Request from a serverkey with openssl
to create a selfsigned certificate from a Certificate Request with openssl

Prerequisits

openssl

Create serverkeys

The first step in certificate creatation is to create serverkeys for a CR (Certificate Request) on your servermachine. That means, your will create a 2048 bit RSA server.key on your local servermachine and reques a cert for for this server on a CA (Certificate Authority) so that the CA can approve this request with a cert for this servermachine key.

Every cert is signed to only one serverkey, except when you are using wildcard certificates

Normally you will use a common CA like Verisign, Commode or Geotrust. In our case we want to create a selfsigned certificate. In that case we are our own CA.

Clearly, thats only usefull, if we want to develop locally or internaly.

So let's create the server.key's

openssl genrsa -des3 -out server.key 2048

At next you will be asked for a passphrase. Type in a secure password with min length 5 letters, a specialchar and 1 digit

That's it. You will now find a file "server.key" in the current directory

Create a server.key for passwordless encryption.
The server.key you have cfreated will ask everytime for a pasword, if you use it. To prevent that, we will now create a further serer,key, which doens't need a password, because it's based on the password encrypted key, we have created before.

openssl rsa -in server.key -out server.key.insecure

This will save a new key "server.key.insecure"

Let's rename the key
To get rid of the insecure word in a serverkey, we simply rename them. :)

mv server.key server.key.secure

mv server.key.insecure server.key

Now we have created our passwordless server,key.

Create a CR for your server.key

Now we have to create our Certificate Request (CR) for our server.key. This can also be done by openssl

openssl req -new -key server.key -out server.csr

At next you will be asked for the server.key passphrase.

In the next prompts you will be asked for Detailinformations of your company.

We want to create a cert for local development under "https://nodejs.local".

So your FQDN is nodejs.local

Don`t forget to add nodejs.local to your local /etc/hosts

After you typed in your password,openssl will save a file "server.csr" in your current directory.

Now, you can send this CertRequest to your CertAuthority or you can self sign the certificate

Selfsign your certificate

For local development we do not want to spend 100 Bugs a year, So lets selfsign the cert. But you will never use that for stage or live / prod system, where your customers are logging them in.

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Great. Now we have a new certificate for our dev environment

You can use the generated "the server.crt" and "server.key" files in your application to en- and decrypt requests

It is a good idea to store the files in a central folder like /etc/ssl/certs and /etc/ssl/keys

At last you have to import this new cert in your browser.

Under Settings->Advanced->Certificates you can inport it under "Certificate Authorities"

Cheers !

webtexx

Dieses Blog durchsuchen

Sonntag, 25. September 2016

node.js: create and scale a simple rest microservice cluster with cluster.js and express to your servers cpu

Prerequisits

Create a npm project

Install needed node libraries

Create index file

Create your routings

Create your cluster

Test your service

http2: make express talk https with spd2

Prerequisits

Get the ssl cert and the server key

Create an node-express / spdy app

Samstag, 24. September 2016

ivy: publish to nexus - unauthorized

ant: retrieve , resolve and publish dependencies to nexus

Prerequisits

Setup the build.xml

Setup the ivy.xml

Setup the ivysettings.xml

Setup the build.properties

Run your build

Dienstag, 20. September 2016

ant: create targets for different os

Sonntag, 18. September 2016

https: use node.js spdy to implement http2

Prerequisits

Get the ssl cert and the server key

Create a simple webapp using http2

openssl: create a self signed certificate for a local host entry

Create serverkeys

Create a CR for your server.key

Selfsign your certificate