Service Assurance/Management

Wednesday, September 12, 2018

Oracle SQLPlus commands

/appl/oracle/app/oracle/product/12.1.0/client_1/bin/sqlplus ncim/ncim@pgv012265:20001/NCIMP_APP.oradb.albertsons.com

./sqlplus reporter/reporter@pgv012266:20001/HISTNCP_APP.oradb.albertsons.com

Tuesday, September 11, 2018

ITNM poller not running as root after set_uid script

Had an issue at customer where the ICMP poller was not running and below warning messages are seen in the log file

2018-09-11T14:37:28: Information: I-MOM-001-013: [641210176t] Connected to broker: 127.0.0.1:1883 using clientId: ncp_poller_4098_BACKSTAGE_P
2018-09-11T14:37:28: Information: I-MOM-001-001: [641210176t] ncp_poller[4098] Version 4.2 (Build 60) 64 bit on linux2x86 becoming Primary
2018-09-11T14:37:28: Information: I-DB-001-018: [328492800t] Polling thread: Database poller is connected to NCIM database 'NCIMP_APP.oradb.albertsons.com' (schema 'ncim')
2018-09-11T14:37:28: Information: I-POL-001-003: [326391552t] Opening plugin SnmpPoller: library name is /appl/esm/tivoli/netcool/precision/precision/platform/linux2x86/lib64/libNcpSnmpPoller.so
2018-09-11T14:37:28: Warning: W-SNM-003-012: [326391552t] Cannot create raw IPv4 socket (error = 1)
2018-09-11T14:37:28: Warning: W-SNM-003-009: [326391552t] Error 1: Operation not permitted
2018-09-11T14:37:28: Warning: W-SNM-003-013: [326391552t] ICMPv4 will be disabled
2018-09-11T14:37:28: Information: I-POL-001-003: [326391552t] Opening plugin Monitor: library name is /appl/esm/tivoli/netcool/precision/precision/platform/linux2x86/lib64/libNcpMonitor.so
2018-09-11T14:37:30: Information: I-DB-001-018: [80545536t] Polling thread: Database poller is connected to NCIM database 'NCIMP_APP.oradb.albertsons.com' (schema 'ncim')

This error is because some server mount points have restrictions on running processes as setuid. look at /etc/fstab and if the mount has nosuid, remove this to fix the issue.

Note: restart of the server is required for the new configuration to take effect.

After restart, the log file shows these lines

2018-09-11T17:02:18: Information: I-MOM-001-013: [3563349824t] Connected to broker: 127.0.0.1:1883 using clientId: ncp_poller_18546_BACKSTAGE_P
2018-09-11T17:02:18: Information: I-MOM-001-001: [3563349824t] ncp_poller[18546] Version 4.2 (Build 60) 64 bit on linux2x86 becoming Primary
2018-09-11T17:02:18: Information: I-DB-001-018: [3250796288t] Polling thread: Database poller is connected to NCIM database 'NCIMP_APP.oradb.albertsons.com' (schema 'ncim')
2018-09-11T17:02:18: Information: I-POL-001-003: [3248695040t] Opening plugin SnmpPoller: library name is /appl/esm/tivoli/netcool/precision/precision/platform/linux2x86/lib64/libNcpSnmpPoller.so
2018-09-11T17:02:18: Information: I-POL-001-003: [3248695040t] Opening plugin Monitor: library name is /appl/esm/tivoli/netcool/precision/precision/platform/linux2x86/lib64/libNcpMonitor.so
2018-09-11T17:02:20: Information: I-DB-001-018: [2327533312t] Polling thread: Database poller is connected to NCIM database 'NCIMP_APP.oradb.albertsons.com' (schema 'ncim')
2018-09-11T17:02:21: Information: I-DB-001-020: [3248695040t] Created handle 1 of (unlimited handles) in pool to DB 'NCIM'
2018-09-11T17:02:21: Information: I-DB-001-020: [3248695040t] Created handle 1 of (unlimited handles) in pool to DB 'POLLDATA'
2018-09-11T17:02:21: Information: I-FAI-001-006: [3248695040t] Operating as the primary ncp_poller in a failover pair

Some info on the mount point options.

I see those two options constantly suggested on the web when someone describes how to mount a tmpfs or ramfs. Often also with noexec but I'm specifically interested in nodev and nosuid. I basically hate just blindly repeating what somebody suggested, without real understanding. And since I only see copy/paste instructions on the net regarding this, I ask here.

This is from documentation:
nodev - Don't interpret block special devices on the filesystem.
nosuid - Block the operation of suid, and sgid bits.

But I would like a practical explanation what could happen if I leave those two out. Let's say that I have configured tmpfs or ramfs(without these two mentioned options set) that is accessible(read+write) by a specific (non-root)user on the system. What can that user do to harm the system? Excluding the case of consuming all available system memory in case of ramfs

Thursday, August 30, 2018

snmpwalk using script

$ head retail1.snmpWorking.Seed.200
10.192.124.196 fw0527a
10.221.200.196 fw9815a
10.221.120.196 fw9783a
10.220.24.196 fw9817a
10.195.220.196 fw9788a
10.218.156.196 fw9811a
10.218.88.196 fw9800a
10.218.72.196 fw9791a
10.226.40.196 fw4152a
10.218.132.196 fw9832a

cat retail1.snmpWorking.Seed.200 | awk '{print "printf " $1 "\"-->\"" ";/bin/snmpwalk -v3 -l authNoPriv -u monitor -a MD5 -A pc15uck5 " $1 " sysObjectID"}' > letsSnmpWalk_v3.sh

cat retail1.snmpWorking.Seed.200 | awk '{print "printf " $1 "\" " $2 "\"\" --> \"" ";/bin/snmpwalk -v3 -l authNoPriv -u monitor -a MD5 -A pc15uck5 " $1 " sysObjectID"}' > letsSnmpWalk_v3-retail1.sh

cat retail1.snmpWorking.Seed.200 | awk '{print "printf " $1 "\" " $2 "\"\" --> \"" ";/usr/bin/snmpwalk -v3 -l authPriv -u meraki -a SHA -A pc15uck5 -x DES -X pc15uck5 " $1 " sysObjectID"}'

/usr/bin/snmpwalk -v3 -l authPriv -u meraki -a SHA -A pc15uck5 -x DES -X pc15uck5 10.244.52.196 sysObjectId

sh /tmp/notworking.snmp.sh > out 2>&1 > /tmp/snmp.result

sh /tmp/notworking.snmp.sh > snmpResult.R3 2>&1

cat retail3.snmpWorking.Seed.200 | awk '{print "printf " $1 "\"-->\"" ";/bin/snmpwalk -v3 -l authNoPriv -u monitor -a MD5 -A pc15uck5 " $1 sysObjectID}' > /tmp/notworking.snmp.sh

SNMP install

# yum install net-snmp-utils
Loaded plugins: fastestmirror, priorities
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
deployed | 2.9 kB 00:00:00
internal | 2.9 kB 00:00:00
pc_repo | 2.5 kB 00:00:00
internal/primary_db | 30 kB 00:00:00
Determining fastest mirrors
* deployed: d1yum.safeway.com
* internal: d1yum.safeway.com
Resolving Dependencies
--> Running transaction check
---> Package net-snmp-utils.x86_64 1:5.7.2-24.el7 will be installed
--> Processing Dependency: net-snmp-libs = 1:5.7.2-24.el7 for package: 1:net-snmp-utils-5.7.2-24.el7.x86_64
--> Processing Dependency: libnetsnmp.so.31()(64bit) for package: 1:net-snmp-utils-5.7.2-24.el7.x86_64
--> Running transaction check
---> Package net-snmp-libs.x86_64 1:5.7.2-24.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================
Installing:
net-snmp-utils x86_64 1:5.7.2-24.el7 deployed 197 k
Installing for dependencies:
net-snmp-libs x86_64 1:5.7.2-24.el7 deployed 747 k

Transaction Summary
===================================================================================================================================================================
Install 1 Package (+1 Dependent package)

Total download size: 943 k
Installed size: 3.3 M
Is this ok [y/d/N]: y
Downloading packages:
(1/2): net-snmp-utils-5.7.2-24.el7.x86_64.rpm | 197 kB 00:00:00
(2/2): net-snmp-libs-5.7.2-24.el7.x86_64.rpm | 747 kB 00:00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 12 MB/s | 943 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 1:net-snmp-libs-5.7.2-24.el7.x86_64 1/2
Installing : 1:net-snmp-utils-5.7.2-24.el7.x86_64 2/2
Verifying : 1:net-snmp-utils-5.7.2-24.el7.x86_64 1/2
Verifying : 1:net-snmp-libs-5.7.2-24.el7.x86_64 2/2

Installed:
net-snmp-utils.x86_64 1:5.7.2-24.el7

Dependency Installed:
net-snmp-libs.x86_64 1:5.7.2-24.el7

Complete!
[root@pgv012245 msing90]# snmpwalk --help
USAGE: snmpwalk [OPTIONS] AGENT [OID]

Version: 5.7.2
Web: http://www.net-snmp.org/
Email: net-snmp-coders@lists.sourceforge.net

Friday, August 24, 2018

Shell Script which uses sendmail

#!/bin/sh
#
# Generic script for forwarding Mail from Netcool/OMNIbus
#
# Parameters to be passed from Netcool/OMNIbus are;
# $1 - @Node
# $2 - @Severity
# $3 - "Message" - mail message header text
# $4 - "user" - user to receive mail
# $5 - @Summary
# $6 - @NodeAlias
#
cat <<EOF > /tmp/tmp.$$
To: $4
Subject: $3

This message refers to node $1 ($6) which has the following problem;

$5

The Severity is $2

Severity Table:
5 - Critical (Tickets)
4 - Major
3 - Minor
2 - Warning
1 - Indeterminate
0 - Clear (Resolution)

Sent by the Netcool/OMNIbus Automation system
EOF

# Find the send mail program: it is in different places on different platforms
for senddir in /usr/sbin /sbin /usr/lib /bin /usr/bin
do
        if [ -x "$senddir/sendmail" ]
        then
                break
        fi
done

# Run send mail
"$senddir/sendmail" $4 < /tmp/tmp.$$

rm /tmp/tmp.$$

Thursday, August 23, 2018

ITNM Startup Script - multiple domains

#!/bin/sh
###############################################################################
#
# chkconfig: 35 92 98
# description: Automatic startup/shutdown script for
# Netcool/Precision for IP Networks
#
### BEGIN INIT INFO
# Provides: ncp
# Required-Start: $network $sshd
# Required-Stop: $network $sshd
# X-UnitedLinux-Should-Start: portmap autofs
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Description: Start the ncp daemon
### END INIT INFO

# File ID: itnm/src/process/templates/ncp_auto.sh, itnm, itnm1.1
# Version: 1.2
# Last Update: 4/20/07 14:00:21
#
# IBM Confidential
# Licensed Materials - Property of IBM
# (c) Copyright IBM Corp. 2007 All Rights Reserved.
#
###############################################################################
#
# To install this in the /etc directory, run ncp_create_control_scripts.sh
# from the installation directory as root.
#
# Usage: ncp start | stop
# Exit status: 0=success, 1=warning, 2=failure
#
###############################################################################
## All comments here

ITNMUSER=netcool
#DOMAINLIST="NCPRD_P"
DOMAINLIST="RETAIL2_P RETAIL1_B"

# These variables are set by 'sed' in the creation script.
set_edited_vars () {
NCHOME=/opt/ibm/tivoli/netcool
ITNM_BIN_DIR=/opt/ibm/tivoli/netcool/precision/bin
ITNM_CONTROL_FUNCS=/opt/ibm/tivoli/netcool/precision/bin/itnm_control_functions.sh
ITNM_AUTO_LOG_FILE=/opt/ibm/tivoli/netcool/log/precision/boot.log
export ITNM_AUTO_LOG_FILE
}

#########################################################################
#
# MAIN
#
#########################################################################

set_edited_vars
. $ITNM_CONTROL_FUNCS
check_exec_exists ${ITNM_BIN_DIR}/ncp_control.sh

# Tell subordinate script that this is called from the /etc/init directory
AUTO_INIT=YES
export AUTO_INIT

if `grep "^admin=" $NCHOME/etc/itnm.cfg 1>/dev/null 2>&1`; then
user=`grep "^admin=" $NCHOME/etc/itnm.cfg 2>/dev/null | cut -f2 -d=`
else
user=`grep "^installer=" $NCHOME/etc/itnm.cfg 2>/dev/null | cut -f2 -d=`
fi

case "$1" in
start|stop)
for DOMAIN in $DOMAINLIST
do
#echo "Starting Domain Name .... " $DOMAIN
su - $user -c "${ITNM_BIN_DIR}/ncp_control.sh -domain $DOMAIN $1"
done
;;
restart|reload)
$0 stop
$0 start
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
esac

exit $?

Wednesday, August 22, 2018

SNMPWALK v3

snmpwalk -v3 -l authNoPriv -u monitor -a MD5 -A pc15uck5 10.251.72.1 sysObjectID
snmpwalk -v3 -l authNoPriv -u monitor -a MD5 -A pc15uck5 172.27.132.28

R3 new - doesn't work
/usr/bin/snmpwalk -v3 -l authNoPriv -u monitor -a MD5 -A pc15uck5 10.244.52.196
snmpwalk -v3 -l authPriv -u meraki -a SHA -A pc15uck5 -x DES -X pc15uck5 10.244.52.196

[netcool@pgv01222f ~]$ /usr/bin/snmpwalk -v3 -l authPriv -u meraki -a SHA -A pc15uck5 -x DES -X pc15uck5 10.244.52.196
snmpwalk: Timeout
[netcool@pgv01222f ~]$

R3 old - works fine
/usr/bin/snmpwalk -v3 -l authNoPriv -u monitor -a MD5 -A pc15uck5 10.244.52.196

Thursday, August 2, 2018

sending snmp traps

yum install net-snmp
yum install net-snmp-utils

# snmptrap -c public -v 2c 172.26.41.189 "" 1.3.6.1.2.1.43.18.2.0.1

# snmptrap -v3 -e 0x111100000000000000000000 -u emsuser -a SHA -A titpw4ems -x DES -X tiabpw4e 172.26.41.189:162 "" coldStart.0

- I have seen mttrapd probe didn't process these alarms because the firewalld was running and blocking the traffic

Thursday, June 7, 2018

Install & Configure - Synthetic Playback agent on APM v8.1.4

Prerequisites for play back agent install.

1. Firefox ESR 52.x.0

[root@apm-nms software]# yum install firefox

[root@apm-nms software]# /bin/firefox -v

Mozilla Firefox 52.8.0

[root@apm-nms software]#

2. set the repo using the command

subscription-manager repos --enable rhel-7-server-optional-rpms

[root@apm-nms agent]# subscription-manager repos --enable rhel-7-server-optional-rpms

Repository 'rhel-7-server-optional-rpms' is enabled for this system.

3. xvfb

yum install Xvfb

Loaded plugins: product-id, search-disabled-repos, subscription-manager

4. install this package for killall command that's being used in the /etc/init.d/xvfb script

yum install psmisc

5. copy this script in /etc/init.d/xvfb

#!/bin/bash

#chkconfig: 345 95 50

#description: Starts xvfb on display 99

if [ -z "$1" ]; then

echo "`basename $0` {start|stop}"

exit

case "$1" in

start)

/usr/bin/Xvfb :99 -screen 0 1280x1024x24 -ac -extension GLX&

;;

stop)

killall Xvfb

;;

esac

6. change the permissions for xvfb

[root@apm-nms agent]# ls -ltr /etc/init.d/xvfb

-rwxr-xr-x. 1 root root 262 Jun 7 13:27 /etc/init.d/xvfb

7. edit the /etc/hosts file and add these 2 entries to the start

127.0.0.1 localhost

10.151.26.14 apm-corp.nms.rcn.net

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

8. export IGNORE_PRECHECK_WARNING=1

9. export SKIP_PRECHECK=1

10. [root@apm-nms APMADV_Agent_Install_8.1.4.0.3]# env|grep SKIP

SKIP_PRECHECK=1

11. [root@apm-nms APMADV_Agent_Install_8.1.4.0.3]# ./installAPMAgents.sh

The following products are available for installation:

1) Monitoring Agent for Cisco UCS

2) Monitoring Agent for DataPower

3) Monitoring Agent for DB2

4) Monitoring Agent for HTTP Server

5) Monitoring Agent for IBM Integration Bus

6) Monitoring Agent for JBoss

7) Monitoring Agent for Linux KVM

8) Monitoring Agent for MongoDB

9) Monitoring Agent for WebSphere MQ

10) Monitoring Agent for MySQL

11) Monitoring Agent for NetApp Storage

12) Monitoring Agent for Nodejs

13) Monitoring Agent for OpenStack

14) Monitoring Agent for Oracle Database

15) Monitoring Agent for Linux OS

16) Monitoring Agent for PHP

17) Monitoring Agent for PostgreSQL

18) Response Time Monitoring Agent

19) Monitoring Agent for Ruby

20) Monitoring Agent for SAP Applications

21) Monitoring Agent for Siebel

22) Monitoring Agent for Synthetic Playback

23) Monitoring Agent for Tomcat

24) Monitoring Agent for VMware VI

25) Monitoring Agent for WebSphere Applications

26) Monitoring Agent for WebLogic

27) Monitoring Agent for WebSphere Infrastructure Manager

28) all of the above

Type the numbers that correspond to the products that you want to install. Type "q" to quit selection.

If you enter more than one number, separate the numbers by a space or comma.

Type your selections here (For example: 1,2): 15,22

The following agents will be installed:

Monitoring Agent for Linux OS

Monitoring Agent for Synthetic Playback

Are your selections correct [ 1-Yes, 2-No; default is 1 ]? 1

Please specify the agent home (default: /opt/ibm/apm/agent):

The agent will be installed in /opt/ibm/apm/agent

Do you accept the license agreement(s) found in the directory /opt/software/APMADV_Agent_Install_8.1.4.0.3/licenses?

Please enter [ 1-to accept the agreement, 2-to decline the agreement ] : 1

User has accepted the license agreement(s).

Monitoring Agent for Linux OS will be installed.

Monitoring Agent for Synthetic Playback will be installed.

Installing Monitoring Agent for Linux OS.

Installing Monitoring Agent for Synthetic Playback.

#####################################################################################################################

The following agents were installed successfully into the /opt/ibm/apm/agent directory:

Monitoring Agent for Linux OS

Agent status:

Agent is running. Process ID is 79114

Server connection status: Connecting

For more information, see the following file: /opt/ibm/apm/agent/logs/lz_ServerConnectionStatus.txt

Monitoring Agent for Synthetic Playback

Agent status:

Agent is not running

The installation log file is /opt/ibm/apm/agent/logs/APMADV_Agents_install.log.

For any agent that is running, the agent is configured with the default settings.

For any agent that is not running, you must configure the agent before it can be started.

To configure your agents, use the following detailed instructions:

For IBM Cloud Application Performance Management SaaS:

http://ibm.biz/kc-ipmcloud-configagent

For IBM Cloud Application Performance Management On Premises:

http://ibm.biz/kc-ipm-configagent

As part of the configuration instructions, you will use the following commands to configure and manage each installed agent:

Monitoring Agent for Linux OS /opt/ibm/apm/agent/bin/os-agent.sh start or stop or status or uninstall

Monitoring Agent for Synthetic Playback /opt/ibm/apm/agent/bin/synthetic_playback-agent.sh config or start or stop or

status or uninstall

#####################################################################################################################

[root@apm-nms APMADV_Agent_Install_8.1.4.0.3]#

Reboot the box

systemctl reboot

Once you have the agent installed the agent will report to the APM server automatically.

Go into APM WEBUI and check for the Linux VM

To configure the synthetic agent on the new server

[root@apm-corp bin]# ./synthetic_playback-agent.sh config

Configuring Monitoring Agent for Synthetic Playback

Edit 'Monitoring Agent for Synthetic Playback' settings? [ 1=Yes, 2=No ] (default is: 1):

The data center name for user to select as playback point of presence.

The data center name for user to select as playback point of presence. (default is: ):corp

This parameter allows you to specify the trace level used by the Java providers.

Java trace level [ 1=Off, 2=Error, 3=Warning, 4=Information, 5=Minimum Debug, 6=Medium Debug, 7=Maximum Debug, 8=All ] (default is: 2):2

A class path that lists jars required by Java API data provider that are not included with the agent. This class path is appended to the class

path generated by the agent.

Class path for external jars (default is: ):

Configuration completed successfully.

STOP the agent

[root@apm-corp bin]# ./synthetic_playback-agent.sh stop

Processing. Please wait...

Stopping Monitoring Agent for Synthetic Playback ...

Monitoring Agent for Synthetic Playback was stopped gracefully.

START

[root@apm-corp bin]# ./synthetic_playback-agent.sh start

Processing. Please wait...

Starting the Monitoring Agent for Synthetic Playback...

Monitoring Agent for Synthetic Playback started

SCP the FIX PACK FOR the playback agent and install it

[root@apm-corp software]# /opt/ibm/apm/agent/bin/synthetic_playback-agent.sh stop

Processing. Please wait...

Stopping Monitoring Agent for Synthetic Playback ...

Monitoring Agent for Synthetic Playback was stopped gracefully.

[root@apm-corp software]# /bin/scp dpatel@207.172.7.36:/opt/software/8.1.4.0-IBM-APM-SYNTHETIC-PLAYBACK-AGENT-XLINUX-IF0003.tar .

dpatel@207.172.7.36's password:

8.1.4.0-IBM-APM-SYNTHETIC-PLAYBACK-AGENT-XLINUX-IF0003.tar 100% 46MB 57.3MB/s 00:00

[root@apm-corp software]#

tar -xvf 8.1.4.0-IBM-APM-SYNTHETIC-PLAYBACK-AGENT-XLINUX-IF0003.tar

[root@apm-corp software]# cd 8.1.4.0-IBM-APM-SYNTHETIC-PLAYBACK-AGENT-XLINUX-IF0003

[root@apm-corp 8.1.4.0-IBM-APM-SYNTHETIC-PLAYBACK-AGENT-XLINUX-IF0003]# ls -ltr

total 20

-rw-r--r--. 1 root root 95 Apr 10 02:17 patch.properties

-rwxr-xr-x. 1 root root 13133 Apr 10 02:17 apmpatch.sh

drwxr-xr-x. 4 root root 85 Jun 7 14:09 patches

[root@apm-corp 8.1.4.0-IBM-APM-SYNTHETIC-PLAYBACK-AGENT-XLINUX-IF0003]# ./apmpatch.sh

Detected IBM Performance Management component(s) installed in /opt/ibm/apm/agent.

The patches for the following components will be installed:

Monitoring Agent for Synthetic Playback [current version: 01000502, patched version: 01000503]

Do you want to continue [ 1-Yes, 2-No; default is "1" ]? 1

Installing patches...

Successfully installed patch for Monitoring Agent for Synthetic Playback.

systmectl reboot

Now the "corp" agent should show up when you go to synthetic script editor

schedule a script

and under location you should have two entries 1. prod and 2. Corp

Friday, May 11, 2018

ITNM Licensing - Net or Client Tracker

TANDBERG-RX8320	80	Client
Cisco C1900	180	Client
Cisco C2900	71	Net Dev
Cisco C2960X	26	Net Dev
Cisco C860	26	Client
Juniper SRX1500	1	Net Dev
Juniper SRX345	33	Net Dev
T-Marc 254	28	Client
T-Marc 280	596	Net Dev
T-Marc 340-E	1	Net Dev
T-Marc 340-F	779	Net Dev
T-Marc 380	824	Net Dev
Cisco 2960	3000	Net Dev