Using Perl to read from elasticsearch

A perl script to read into elasticsearch

use Search::Elasticsearch; use URI::Escape; use DateTime;

$dt = DateTime->now; $start_timestamp = join ' ', $dt->ymd, '00:00:00'; $end_timestamp = join ' ', $dt->ymd, '23:59:59';

my $client = "something";

my $es = Search::Elasticsearch->new(trace_to => ['File','/var/log/perl/log-'.$start_timestamp.'.log'],nodes=>['http://10.9.8.x:9200/']);

my $scroll = $es->search(index => 'logstash-*',body => {"_source" => ["Name","syslogHostName"],"query" => { "match" => { "ClientName.raw" => "$client" } } }, size => 3000);

my @results = @{ $scroll->{hits}{hits} }; print "Total number of hosts: ".scalar @results."\n\n"; for (my $i=0 ; $i < (scalar @results); $i++ ) { print $results[$i]->{_source}->{syslogHostName}."\n"; }

Elasticsearch cluster setup - Three node cluster

The following changes are required in elasticsearch.yml

Node1

cluster.name: lab

node.name: node-name

bootstrap.memory_lock: true

bootstrap.system_call_filter: false

network.host: hostname

http.port: 9200

node.master: true

node.data: true

discovery.zen.ping.unicast.hosts: ["x.x.x.x", "x.x.x.x"]

discovery.zen.minimum_master_nodes: 2

Node2

cluster.name: lab

node.name: node-name

bootstrap.memory_lock: true

bootstrap.system_call_filter: false

network.host: hostname

http.port: 9200

node.master: true

node.data: true

discovery.zen.ping.unicast.hosts: ["x.x.x.x", "x.x.x.x"]

discovery.zen.minimum_master_nodes: 2

Node3

cluster.name: lab

node.name: node-name

bootstrap.memory_lock: true

bootstrap.system_call_filter: false

network.host: hostname

http.port: 9200

node.master: true

node.data: true

discovery.zen.ping.unicast.hosts: ["x.x.x.x", "x.x.x.x"]

discovery.zen.minimum_master_nodes: 2

Further advice to setup coordination node in the cluster

It is not feasible to have a coordination node(client node) in a 4 nodes cluster. To have such provision we need to have 5 nodes cluster; I will explain the same with the following details.

The master/data node architecture is configured with three important parameters they are:

1. node.master
2. node.data
3. discovery.zen.minimum_master_nodes

The first two parameters say whether a node is master eligible or not by setting node.master to true. The third parameter is an important factor to elect a new master in case the acting master goes down.

If a cluster has three eligible master nodes then the value of minimum_master_nodes is calculated as (3/2)+1 = 2. So in a 4 node cluster there should be four master eligible nodes and the minimum_master_nodes value should be equal to 3. In that case we cannot have a coordination node which should not be a master eligible node. It is always a recommended practice to have the number of nodes in odd series than an even series. So if we consider 5 nodes cluster then we will have four master eligible nodes and one coordination node (5/2)+1 = 3.

Significance of discovery.zen.minimum_master_nodes:

If a master goes down in a cluster this value governs the election of new master. Unless the value is met, for example if the value is three unless there are three master eligible nodes a new master will not be elected. This is to avoid a split brain issue. A split brain problem may occur if any of the data nodes goes out of cluster due to a network outage for example, then it will not promote itself to become master because there is only one master eligible node. If this is not controlled then the node which is not connected will promote itself as a master and it will cause data loss when put back to the cluster. To avoid such issues this value is considered significant.

So a 5 nodes cluster can be organized as

1. Master/Data node – a primary master and master eligible node
2. Master/Data node – a master eligible node
3. Master/Data node – a master eligible node
4. Master/Data node – a master eligible node
5. Coordination(client) node – not a master eligible node

The kibana and logstash can be connected to this coordination node which will act as a load balancer for the elasticsearch cluster.

Chef administration

Backup and restore
chef-server-ctl backup --yes
it will bring down chef server and then take backup

chef-server-ctl restore /path/to/backup

Docker notes

To start a service when a container starts,

use entrypoint

ENTRYPOINT service elasticsearch start && bash

A sample knife.rb file with exception for ssl mode

current_dir = File.dirname(__FILE__)

log_level                :info

log_location             STDOUT

node_name                'rajagopalan'

client_key               '/root/chef-repo/.chef/rajagopalan.pem'

validation_client_name   'hexaware'

validation_key           '/root/chef-repo/.chef/hexaware-validator.pem'

chef_server_url          'https://api.chef.io/organizations/ORG_NAME'

cache_type               'BasicFile'

cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )

cookbook_path            ['#{current_dir}/../cookbooks']

Vi  ~/.gemrc

Add this line to bypass ssl check

:ssl_verify_mode: 0

Add this line to knife.rb to exclude ssl check while executing knife ec2 server create

Excon.defaults[:ssl_verify_peer] = false

Fix - ERROR: Server returned error 500 for https://127.0.0.1/users/ - Chef

If the following error is faced in chef-server, version 12, then do the following to fix the issue.


ERROR: Server returned error 500 for https://127.0.0.1/users

open the file /opt/opscode/embedded/cookbooks/private-chef/templates/default/oc_erchef.config.erb in vi editor and go to line 220.

Replace the following line :

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

with

{s3_url, "https://private-chef.opscode.piab:4000"},

and then run chef-server-ctl reconfigure.

Reason and Solution:

nginx will listen on port 4000 for HTTPS connections and not the default port of 443.

During cookbook uploads, the opscode-erchef service talks to bookshelf via the s3_url in its configuration file (/var/opt/opscode/opscode-erchef/etc/app.config). This configuration file is rendered via a template(opscode-omnibus/files/private-chef-cookbooks/private-chef/templates/default/oc_erchef.config.erb), a portion of which looks like:

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},
Thus, the rendered configuration file will have an s3_url like:

{s3_url, "https://private-chef.opscode.piab"},
Given this configuration, erchef will attempt to contact erchef on port 443, the default HTTPS port. Unfortunately, nothing is listening on 443, the request to bookshelf fails and erchef returns a 500 to the user.

An astute user may attempt to set bookshelf['vip'] in private-chef.rb to something like:

bookshelf['vip'] = 'private-chef.opscode.piab:4000'

Reference : https://github.com/chef/chef-server/issues/50

Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused

run mongod process with the dbpath parameter

mongod --dbpath /home/mongo/data/db

create the path if it does not exits.

Jenkins scp plugin - can't connect to server issue, Jenkins scp repositories - can't connect to server, SEVERE: Algorithm negotiation fail

The issue can be resolve, by opening the /etc/ssh/sshd_config file and add the following line:

KexAlgorithms diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

save and then restart ssh server: service ssh restart.

The problem is fixed.

Using proxy for apt-get

When your server is behind a proxy, you can do the following steps to run apt-get through a proxy:

open the file /etc/apt/apt.conf, if not create it and add the lines

Acquire::http::proxy "http://xxx\username:password@proxy.web.local:/3128";

save and exit. Now everything will go fine.

PostgreSQL PITR

PostgreSQL allows you to restore the database to a specific point in time by using

one of the three options:

recovery_target_name,recovery_target_time,recovery_target_xid,pause_at_recovery_target

Go to master node and do the following steps

Edit postgresql.conf and add\edit  the following lines as given below

listen_addresses = '*'

port = 5432

max_connections = 100

wal_level = hot_standby

archive_mode = on

archive_command = 'cp %p /usr/local/pgsql/archive/%f'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

For demo purpose we install two instance in the same host on two different ports.

Install the second instance on the port 5433

Open the pg_hba.conf in master and add these lines at the bottom

host    all     all     0.0.0.0/0       trust

host    replication     all     0.0.0.0/0       trust

host    replication     postgres        0.0.0.0/0       trust

Start the postgresql on port 5432 and create a test database and connect to it

create database test_pitr;

\c test_pitr

 create table test(col1 int, col2 char(10));

insert into test select generate_series(1,100),'test';

Go to data directory of secondary node and clear it,

 cd /usr/local/pgsql2/data

 rm -rf *

Connect  to secondary node at 5433 and execute pg_basebackup, which will take care of pg_start_backup('label'); and pg_stop_backup();

pg_basebackup -D /usr/local/pgsql2/data/ --write-recovery-conf --xlog-method=fetch --verbose -h localhost

This command will copy data folder from master with recovery.conf

You will get output something like  this

transaction log start point: 4/8D000028 on timeline 1

transaction log end point: 4/8E000050

pg_basebackup: base backup completee

Connect to master node on 5432 and execute the command to create restore point. The point till which the wal log will be replayed

 select pg_create_restore_point('patch_2015_09_17_2');

Now create some more records

insert into test select generate_series(101,1000),'test';

After this the data directory from master will be copied over to secondary. Open the postgresql.conf and edit as given below

hot_standby = on

And leave the rest as same

Open recovery.conf and edit it as below

standby_mode = 'on'

primary_conninfo = 'user=postgres host=localhost port=5432'

pause_at_recovery_target = true

recovery_target_name = patch_2015_09_17_02

Start the server on 5433

If we check the log file, these are the important sections:

entering standby mode..

recovery stopping at restore point "patch_of_2014_07_02", time

2014-07-02 12:08:57.507946+05:30

recovery has paused

Execute pg_xlog_replay_resume() to continue

Connect to psql on the secondary node and check the record

\c test_pitr

 select * from test order by col1 desc limit 2;

Try to create a table and you may find this error,

ERROR: cannot execute CREATE TABLE in a read-only transaction

The database is still in read-only mode.

Execute the following command

 SELECT pg_xlog_replay_resume();

It's done

Modules got blocked in powershell v1.0

The best way to unblock in v1.0 is to remove the module from powershell location
eg: C:\windows\system32\windowspowershell\v1.0\modules\pswindowsupdate

and extract the downloaded module again into the same location.

How to move putty session from one computer to another

http://mewbies.com/how_to_export_remove_putty_data_saved_sessions_tutorial.htm

Mysql replication failure

when faced with mysql replication failure problem,
"Please check the replication failure from Master Host"

do the following steps,
connect to the slave machine, open command prompt, it is cmd in windows or go to terminal in linux

connect to mysql as "mysql -u username -p", when prompt for password, enter the password
issue the command, "show slave status \G" and check for the errors.  Check the parameter which tells how long the slave is behind the master.

Do then,
1. stop slave;
2. start slave;
3. show slave status \G. Again check for erros and no of seconds the slave is behind the master

Even if this doesn't work try restarting the mysql server in slave machine, in worst case try doing it in master and then repeat the above steps

If any of the mysql table is crashed, then execute the following command as

repair table db.tablename; 

Postgresql streaming replication

pg_dump: Error message from server: ERROR: canceling statement due to conflict with recovery

As I have worked and came to know that, i feel, tweaking with the option max_standby_streaming_delay > 60 secs will be solving the problem

Postgresql table partitioning

Error code 4064 - Cannot Open User Default Database (error 4064) - MSDN - Microsoft

This issue can be solve with the following command

sqlcmd -d master -U sa -P mypassword
and then:
alter login sa with default_database = master

Command prompt to backup and restore

sqlcmd -U username -S servername -Q "BACKUP DATABASE dbname TO DISK='path\filename.bak'"

Streaming Replicaiton in Postgresql

A short information on streaming replication in postgresql 9.2
Requesties
The postgresql servers both master and slave, should be of the same configuration (i.e., the data folder should be in the same location for slave as the master).
Master server changes:
open postgresql.conf master and do the changes
listen_addresses = "*"
wal_level = hot_standby
archive_mode = on
archive_command = "copy %p \\\\192.168.1.1\\wal_archive\\%f"
archive_timeout = 3600
wal_senders = 5
wal_keep_segments = 32

pg_hba.conf
host    all    all    0.0.0.0/0    trust
host    replication   all  0.0.0.0/0   trust

In the slave server, either remove the existing data folder or rename it to data_old, coming back to master issue the following command
pg_basebackup -U postgres -p 5432  -D  "\\192.168.1.1\PostgreSQL\9.2\data"
Windows:
"C:\\Program Files(x86)\PostgreSQL\9.2\bin\pg_basebackup" -U postgres -p 5432 -D "\\192.168.1.1\c$\Program Files(x86)\PostgreSQL\9.2\data"

Use the -w attribute to make your shell script to work uninterrupted for passwords, with the help of pgpass.conf file.

Slave Server Changes
After the files are being copied from  master server, open the postgresql.conf file and comment out all the settings and have only the below change
hot_standby = on

After this prepare a recovery.conf file, open a notepad and add the following code and save it as recover.conf in the data folder of the slave server, where its postgresql.conf file exists
standby_mode = on
primary_conninfo = 'host=192.168.1.0 port = 5432 user = postgres password = sydney11'
trigger_file = 'C:\\trigger_file.trigger'
restore_command = 'copy C:\\wal_archive\\%f %p"

Working
The master will continuously send the wal files to the shared folder of the slave server.
the slave will restore the received wal files using the restore_command from the recovery.conf file
when the master is down create a trigger file, as given in the recovery.conf
the presence of trigger file will alert the slave that the master is down and change the recovery.conf as recovery.done and make it as a master server.

Tamizh numeric system