Aku mendapat tambahan pekerjaan dari ngurusin Security, sekarang menjadi Server Operation Leader di perusahaan operator WiMax di Jakarta.  Mengerikan melihat banyaknya server didalam data centre dan secara expert technical on site  hanya saya yang  dibebani tanggung jawab.  Manager dan Head operation sama sekali tidak bisa mengerti detail hal-hal teknis , taunya beres, bingung saya model rekruitment disini, menurut kacamata pengalaman saya di bidang IT sudah sewajarnya Manager dan Head memiliki basic kuat hal-hal teknis yang berada dibawah tanggung jawabnya, jika staffnya tidak available maka sang Manager dan Head bisa turun gunung membenarin server.

But show must go on, saya enjoy saja, kembali ke splunk+syslog-ng, rencana saya memasukkan log pada 16 server Solaris, 20 server Linux(Red Hat Enterprise Linux 5 & Debian 6), 5 server Windows Server 2003 ke dalam server centralized log ini. Departement lain yaitu IP/Network Core juga meminta agar device cisco dan perangkat mereka juga bisa masuk ke dalam centralized log ini. Melihat hal tersebut saya harus punya senjata yang dapat menjadi bukti letak permasalahan(root causes) sebuah problem. Based pengalaman dari kantor sebelumnya saya tambahkan sebuah fitur baru untuk NOC Department yaitu Log terpusat(Centralized loging) maksudnya semua catatan log pada server dan device cisco dapat disimpan dalam server centralized ini.

Berikut proses instalasi dan konfigurasi instalasi Splunk+Syslog-ng dan konfigurasi syslog pada server Linux,Solaris,Windows Server

 

Setelah server dan IP saya dapatnya, saya install dengan sistem operasi Linux distribusi Debian 6, dan Download Splunk di : http://www.splunk.com/download

sitra-splunk:/home/gtoms# ls

splunk-4.1.6-89596-linux-2.6-intel.deb

sitra-splunk:/home/gtoms# dpkg -i splunk-4.1.6-89596-linux-2.6-intel.deb

Selecting previously deselected package splunk.

(Reading database … 91122 files and directories currently installed.)Unpacking splunk (from splunk-4.1.6-89596-linux-2.6-intel.deb) …

Setting up splunk (4.1.6-89596) …

Splunk has been installed in:        /opt/splunk
To start Splunk, run the command:        /opt/splunk/bin/splunk start
To use the Splunk Web interface, point your browser at:        http://sitra-splunk:8000
Complete documentation is at http://www.splunk.com/r/docs—————–

sitra-splunk:/home/gtoms# cd /opt/splunk

sitra-splunk:/opt/splunk# ls

bin  etc  ftr  include  lib  license-eula.txt  openssl  README-splunk.txt  share  splunk-4.1.6-89596-Linux-i686-manifest

sitra-splunk:/opt/splunk# dpkg –status splunk

Package: splunk

Status: install ok installed

Priority: extraSection: non-freeMaintainer: Splunk Inc.Architecture: i386Version: 4.1.6-89596Description: Splunk Copyright: 2005-2010 Splunk Inc. Splunk is the IT Search engine.

Developer API, Example Modules, or Your Extensions (including but not limited to offering the functionality of the Free Software on an applications service provider or time sharing basis), except as expressly permitted in the Splunkbase Application Developer Agreement; (iv) decompile, disassemble or reverse-engineer the Software or otherwise attempt to derive the Software source code; (v) disclose to any third party the results of any benchmark tests or other evaluation of the Software, or (vi) authorize any third parties to do any of the above.
OWNERSHIP. Splunk and/or its licensors own all worldwide right, title and interest in and to the Software, including all worldwide intellectual property rights therein. You will not delete or in any manner alter the copyright, trademark, and other proprietary rights notices appearing in or on the Software as provided. All right, title, and interest in and to all copies the Splunk Developer API, and the Example Modules remains with Splunk and/or its licensors. The Software, Splunk Developer API, and Example Modules are copyrighted and protected by the laws of the United States and other countries, and international treaty provisions. You may not remove any copyright notices from the Software, the Splunk Developer API, or the Example Modules.
PURCHASED SOFTWARE LICENSE AND FEES. In order to access and use the Software, you are required to pay to Splunk the License Fees in accordance with your Order Confirmation. The License Fees will be due and payable in accordance with the terms set forth in your Order Confirmation. Any failure to pay the License Fees in accordance with an Order Confirmation will result in automatic revocation and termination of this Agreement and all rights and licenses granted hereunder. All License Fees are non-refundable once paid.
MAINTENANCE AND SUPPORT. Subject to your payment of the applicable annual maintenance and support fees set forth in your Order Confirmation (the “Support Fees”), Splunk will provide the level of support for the Purchased Software identified in your Order Confirmation in accordance with the support descriptions set forth on Splunk’s website at www.splunk.com. Splunk is not obligated to support, update or upgrade the Free Software.
PURCHASED SOFTWARE VERIFICATION AND AUDIT. At Splunk’s written request, you will furnish Splunk with a certification signed by an officer of your company verifying that the Software is being used in accordance with the terms and conditions of this Agreement and the applicable Order Confirmations. Upon at least ten (10) days prior written notice, Splunk may audit your use of the Software to ensure that you are in compliance with the terms of this Agreement and the applicable Orders. Any such audit will be conducted during regular business hours at your facilities, will not unreasonably interfere with your business activities and will be in compliance with your reasonable security procedures. You will provide Splunk with access to the relevant records and facilities. If an audit reveals that you have exceeded the daily peak volume during the period audited, then Splunk will invoice you, and you will promptly pay Splunk any underpaid fees based on Splunk’s price list in effect at the time the audit is completed. If the daily peak volume usage exceeds ten percent (10%) of the licensed usage, then you will also pay Splunk’s reasonable costs of conducting the audit.
PURCHASED SOFTWARE WARRANTY. Splunk warrants that for a period of thirty (30) days after your registration of the Software with Splunk, the Software will substantially achieve any material function described in documentation for the Software published by Splunk. As Splunk’s sole liability and your sole remedy for any failure of the Software to conform to this warranty, Splunk will repair or replace (at Splunk’s option) your copy of the Software.
WARRANTY DISCLAIMER. EXCEPT AS SET FORTH ABOVE, SPLUNK DISCLAIMS ANY AND ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, QUIET ENJOYMENT AND WARRANTIES ARISING OUT OF COURSE OF DEALING OR USAGE OF TRADE. Splunk does not warrant (i) that the Software, developer’s API’S or example modules will meet your requirements, (ii) that the Software will operate in the combinations that you may select, (iii) that the Software will serve the purposes intended by you, or (iv) that the operation of the Software will be error free or uninterrupted or that any Software errors will be corrected.
LIMITATION OF LIABILITY. SPLUNK’S TOTAL CUMULATIVE LIABILITY TO YOU, FROM ALL CAUSES OF ACTION AND ALL THEORIES OF LIABILITY, WILL BE LIMITED TO AND WILL NOT EXCEED THE AMOUNTS PAID BY YOU TO SPLUNK IN THE TWELVE MONTHS PRIOR TO THE EVENT GIVING RISE TO SUCH LIABILITY. IN NO EVENT WILL SPLUNK BE LIABLE TO YOU FOR ANY SPECIAL, INDIRECT, INCIDENTAL, CONSEQUENTIAL OR PUNITIVE DAMAGES (INCLUDING LOSS OF USE, DATA, OR PROFITS, BUSINESS INTERRUPTION, OR COSTS OF PROCURING SUBSTITUTE SOFTWARE) ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE USE OR PERFORMANCE OF THE SOFTWARE, WHETHER SUCH LIABILITY ARISES FROM CONTRACT, WARRANTY, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, AND WHETHER OR NOT SPLUNK HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH LOSS OR DAMAGE. THE PARTIES HAVE AGREED THAT THESE LIMITATIONS WILL SURVIVE AND APPLY EVEN IF ANY REMEDY IS FOUND TO HAVE FAILED OF ITS ESSENTIAL PURPOSE. WITHOUT LIMITING THE FOREGOING, SPLUNK WILL HAVE NO LIABILITY OR RESPONSIBILITY FOR ANY BUSINESS INTERRUPTION OR LOSS OF DATA ARISING FROM THE AUTOMATIC TERMINATION OF THE LICENSE RIGHTS GRANTED HEREIN AND ANY ASSOCIATED CESSATION OF THE SOFTWARE FUNCTIONS. BECAUSE SOME STATES OR JURISDICTIONS DO NOT ALLOW LIMITATION OR EXCLUSION OF CONSEQUENTIAL OR INCIDENTAL DAMAGES, THE ABOVE LIMITATION MAY NOT APPLY TO YOU.
PURCHASED SOFTWARE INDEMNITY. Splunk will defend, indemnify and hold you harmless from and against any loss, damage, liability or cost (including reasonable attorneys’ fees) resulting from any third party claim that the Purchased Software infringes or violates any third party’s patent, copyright or trademark rights; provided that you promptly notify Splunk in writing of any and all such claims. In the event of any loss, damage, liability or cost for which Splunk is obligated to indemnify you hereunder, Splunk shall have sole control of the defense and all related settlement negotiations, and you shall reasonably cooperate with Splunk in the defense and/or settlement thereof at Splunk’s expense; provided that you may participate in such defense using your own counsel, at your own expense.
TERMINATION. You may terminate this Agreement at any time by destroying or returning to Splunk all copies of the Software, including any documentation, in your possession and control, and providing to Splunk a written statement signed by an authorized representative of your company notifying Splunk that you are terminating the Agreement and certifying such destruction or return. Upon thirty days notice, Splunk may terminate this Agreement (and your license rights) upon notice in the event that you breach any provision of this Agreement and have not cured the breach during such notice period. Upon any expiration or termination of this Agreement, the rights and licenses granted hereunder will automatically terminate, and you agree to immediately cease using the Software and to return or destroy all copies of the Software in your possession or control. In the event of termination of this Agreement, Splunk will have no obligation to refund any License Fees, Support Fees, or other fees received from you during the Term. All provisions of this Agreement related to disclaimers of warranties, limitation of liability, remedies, damages, or Splunk’s proprietary rights shall survive termination.
SEVERABILITY. All rights and remedies, whether conferred hereunder or by any other instrument or law, will be cumulative and may be exercised singularly or concurrently. Failure by either Splunk or You to enforce any term will not be deemed a waiver of future enforcement of that or any other term. The terms and conditions stated herein are declared to be severable. Should any term(s) or condition(s) of this Agreement be held to be invalid or unenforceable the validity, construction and enforceability of the remaining terms and conditions of this Agreement shall not be affected.
EXPORT. You agree to comply fully with all relevant export laws and regulations of the United States (“Export Laws”) to ensure that the Software is not (i) exported or re-exported directly or indirectly in violation of Export Laws; or (ii) intended to be used for any purposes prohibited by the Export Laws, including but not limited to nuclear, chemical, or biological weapons proliferation.
GOVERNMENT RESTRICTED RIGHTS. The Software shall be classified as “commercial computer software” as defined in the applicable provisions of the Federal Acquisition Regulation (the “FAR”) and supplements thereto, including the Department of Defense (DoD) FAR Supplement (the “DFARS”). The parties acknowledge that the Software was developed entirely at private expense and that no part of the Software was first produced in the performance of a Government contract. If the Software is supplied for use by DoD, the Software is delivered subject to the terms of this Agreement and in accordance with DFARS 227.7202-1(a) and 227.7202-3(a) (1995), with restricted rights in accordance with DFARS 252.227-7013(c)(1)(ii) (OCT 1988), as applicable. If the Software is supplied for use by a Federal agency other than DoD, the Software is restricted computer software delivered subject to the terms of this Agreement and FAR 12.212(a) (1995); (ii) FAR 52.227-19; or FAR 52.227-14(ALT III), as applicable.
PUBLICITY.  You agree that Splunk may identify you as a Splunk customer on Splunk websites, client lists, press releases, and/or other marketing.  You also agree that Splunk may publish a brief description highlighting your deployment of the Software.
GENERAL. This Agreement shall be governed by and construed in accordance with the laws of the State of California, as if performed wholly within the state and without giving effect to the principles of conflict of law. Any legal action or proceeding arising under this Agreement will be brought exclusively in the federal or state courts located in the Northern District of California and the parties hereby consent to personal jurisdiction and venue therein. If any portion hereof is found to be void or unenforceable, the remaining provisions of this Agreement shall remain in full force and effect. Neither party may assign this Agreement, in whole or in part, except in connection with an internal reorganization or a sale of the business with which this Agreement is associated without Splunk’s prior written consent, and any attempt to assign this Agreement other than as permitted above will be null and void. This Agreement is intended for the sole and exclusive benefit of the parties and is not intended to benefit any third party. Only the parties to this Agreement may enforce it. This Agreement and any Order Confirmations constitute the complete and exclusive understanding and agreement between the parties regarding their subject matter and supersede all prior or contemporaneous agreements or understandings, written or oral, relating to their subject matter. Any waiver, modification or amendment of any provision of this Agreement will be effective only if in writing and signed by duly authorized representatives of both parties.

Do you agree with this license? [y/n]:  y

Copying ‘/opt/splunk/etc/myinstall/splunkd.xml.cfg-default’ to ‘/opt/splunk/etc/myinstall/splunkd.xml’.

Copying ‘/opt/splunk/etc/openldap/ldap.conf.default’ to ‘/opt/splunk/etc/openldap/ldap.conf’./opt/splunk/etc/auth/audit/private.pem/opt/splunk/etc/auth/audit/public.pem[‘openssl’, ‘genrsa’, ‘-out’, ‘/opt/splunk/etc/auth/audit/private.pem’, ‘1024’]/opt/splunk/etc/auth/audit/private.pem generated./opt/splunk/etc/auth/audit/public.pem generated.

Generating RSA private key, 1024 bit long modulus………………………….++++++…..++++++e is 65537 (0x10001)writing RSA key
/opt/splunk/etc/auth/distServerKeys/private.pem/opt/splunk/etc/auth/distServerKeys/trusted.pem[‘openssl’, ‘genrsa’, ‘-out’, ‘/opt/splunk/etc/auth/distServerKeys/private.pem’, ‘1024’]/opt/splunk/etc/auth/distServerKeys/private.pem generated./opt/splunk/etc/auth/distServerKeys/public.pem generated.Generating RSA private key, 1024 bit long modulus……………………….++++++…………….++++++e is 65537 (0x10001)writing RSA key

This appears to be your first time running this version of Splunk.Moving ‘/opt/splunk/share/splunk/search_mrsparkle/modules.new’ to ‘/opt/splunk/share/splunk/search_mrsparkle/modules’.        Creating: /opt/splunk/var/lib        Creating: /opt/splunk/var/run/splunk        Creating: /opt/splunk/var/run/splunk/upload        Creating: /opt/splunk/var/spool/splunk        Creating: /opt/splunk/var/spool/dirmoncache        Creating: /opt/splunk/var/lib/splunk/authDb        Creating: /opt/splunk/var/lib/splunk/hashDb        Checking databases…        Validated databases: _audit, _blocksignature, _internal, _thefishbucket, history, main, sample, summary
Splunk> Take the sh out of IT.
Checking prerequisites…

Checking http port [8000]: open

Checking mgmt port [8089]: open

Checking configuration…  Done.

Checking index directory…  Done.

Checking databases…        Validated databases: _audit, _blocksignature, _internal, _thefishbucket, history, main, sample, summaryAll preliminary checks passed.

Starting splunk server daemon (splunkd)… Done.

Starting splunkweb… /opt/splunk/share/splunk/certs does not exist. W

ill createGenerating certs for splunkweb serverGenerating a 1024 bit RSA private key……………………..++++++……………………………………….++++++writing new private key to ‘privkeySecure.pem’—–Signature oksubject=/CN=sitra-splunk/O=SplunkUserGetting CA Private Keywriting RSA keyDone.
If you get stuck, we’re here to help.Look for answers here: http://www.splunk.com/base/Documentation

The Splunk web interface is at http://sitra-splunk:8000
sitra-splunk:/opt/splunk#

sitra-splunk:/opt/splunk# netstat -nltup |grep splunkd

tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      3155/splunkd

Install Syslog-NG

sitra-splunk:/home/gtoms#apt-get install syslog-ng

sitra-splunk:/home/gtoms#cd /etc/syslog-ng

sitra-splunk:/etc/syslog-ng#vi syslog-ng.conf

options {        chain_hostnames(0);        time_reopen(10);        time_reap(360);        log_fifo_size(2048);        create_dirs(yes);        perm(0640);        dir_perm(0755);        use_dns(no);        stats_freq(0);};

source s_all {        internal();        unix-stream(“/dev/log”);        file(“/proc/kmsg” log_prefix(“kernel: “));        udp();};

filter f_windows {        program(MSWinEventLog);};

filter f_cisco_pix {        host(10.3.10.36);};

filter f_not_others {        not host(10.3.10.36)        and not program(MSWinEventLog);};

destination d_windows {        pipe(“/var/log/buffers/windows”);};

destination d_cisco {        pipe(“/var/log/buffers/cisco”);};
destination d_gen_fifo {        pipe(“/var/log/buffers/syslog”);};
destination d_all {        file(“/var/log/arch/$MONTH$DAY$YEAR”);};

log {        source(s_all);        filter(f_windows);        destination(d_windows);};
log {        source(s_all);        filter(f_cisco_pix);        destination(d_cisco);};
log {        source(s_all);        filter(f_not_others);        destination(d_gen_fifo);};

log {        source(s_all);        destination(d_all);};

Membuat direktory mkdir/mkfifo :

sitra-splunk:/etc/syslog-ng#mkdir /var/log/arch

sitra-splunk:/etc/syslog-ng#mkdir /var/log/buffers

sitra-splunk:/etc/syslog-ng#mkfifo /var/log/buffers/windows

sitra-splunk:/etc/syslog-ng#mkfifo /var/log/buffers/cisco

sitra-splunk:/etc/syslog-ng#mkfifo /var/log/buffers/syslog

Restart syslog-ng

sitra-splunk:/etc/syslog-ng#/etc/init.d/syslog-ng restart

Cek log files :

sitra-splunk:/etc/syslog-ng#cd /var/log/arch

sitra-splunk:/var/log/arch#

sitra-splunk:/var/log/arch# ls

01312011  02052011  02102011  02152011  02202011  02252011  03022011  03072011  03122011  03172011  03222011  03272011  04012011  0406201102012011  02062011  02112011  02162011  02212011  02262011  03032011  03082011  03132011  03182011  03232011  03282011  04022011  0407201102022011  02072011  02122011  02172011  02222011  02272011  03042011  03092011  03142011  03192011  03242011  03292011  04032011  0408201102032011  02082011  02132011  02182011  02232011  02282011  03052011  03102011  03152011  03202011  03252011  03302011  04042011  0409201102042011  02092011  02142011  02192011  02242011  03012011  03062011  03112011  03162011  03212011  03262011  03312011  04052011  04102011

Cek FIFO Buffers

sitra-splunk:/var/log/arch# cat /var/log/buffers/syslog

Cara melempar syslog server atau device lain ke Server Centralized :

Windows Log :

• SNARE Agent• Converts Event Logs to Syslog

UNIX log :

• Menggunakan syslog dan configure syslog  *.* @Syslog Server

Devices Cisco dll :

-Tinggal mengarahkan fasilitas logging pada devices ke IP atau hostname Centralized server.