____________
How to Report
an Incident
If
you learn that your system has been infected, don't turn
off your computer. Instead, quit using the machine, quarantine
it immediately (leave it on but put a sign on the monitor
to warn against further use), and report
the incident to your system administrator, your designated
technical support person,
or the help desk (help@lbl.gov
or x4357). That person can evaluate the situation and, if
appropriate, report the incident to the "Computer Protection
Program Manager" (CPPM@lbl.gov).
If you believe a Lab-site computer you are using has
been infected and you
have not yet been able to contact the appropriate technical
support, you can use the Emergency
Incident Reporting Form, which will send your report directly
to a security engineer via pager, as well as to the CPPM.
It
is important to report attacks right away as a security engineer
can often uncover information that will help prevent future
attacks. However, killing processes, rebooting, etc., can
destroy those clues. Even if the incident seems insignificant,
it could be an important clue to a more widespread attack.
If
for any reason you are unable to contact any of the above
individuals and you believe your computer has been attacked,
you should physically disconnect the computer from the network until a security engineer can look at it. Again, do not shut
the computer down, kill any processes, or unplug the machine
- if at all possible.
24/7 Emergency Contact
Any LBNL related cyber security incident can be reported via the LBNL CPPM Emergency Response Telephone Line 24 hours a day, 365 days a year. This line is staffed during normal business hours, and will activate an emergency pager during non-business hours. Other contact options are listed on the web page: http://www.lbl.gov/ITSD/Security/people/emergency.html
Incident Definition
A cyber incident is defined as any event that is initially perceived to represent a threat to the Lab, involves computers, and requires Lab resources to resolve. Examples are computer attacks, compromises, and worm infections. An incident can be a "false positive" where no actual threat is present, but an investigation is needed to reach that conclusion.
Incident Handler
Except in extraordinary cases that require additional resources, each incident will be handled by a designated Incident Handler. This individual will have primary responsibility for the lifecycle of the incident, as described below. The Incident Handler will generally be a team member of LBNL's direct cybersecurity staff or that of an enclave, and will often be assisted by other cybersecurity personnel. The term ‘Incident Handler' in the remainder of this document has the sense indicated in this paragraph.
Initial Information Gathering and Analysis
When an incident occurs, the Incident Handler will make an initial assessment of the severity of the incident. As the primary responsibility of LBNL Incident Response is to protect LBNL cyber resources, this initial assessment will guide triage efforts. When any initial threat is contained, detailed information about the incident will be gathered and analyzed. A useful technique, which has been developed and refined, is to retrieve from the bulk record traces all network traffic between the targeted systems, and the attacking hosts. Other sources of information may include connection logs, system logs, post mortem data, and reports from other sites.
While the art of incident analysis is outside the scope of this document, a number of questions, some obvious, need to be answered:
• What type of attack was it?
• Was the attack successful?
• How many systems may be involved?
• Were root or administrative privileges acquired?
• How many other accounts may have been compromised?
• What was the sequence of events?
• What, if any, actions were taken by local administrators?
• What is the current level of risk?
• Is the system being attacked a critical system or network component?
• What is the potential or actual rate of incident spreading?
• Does the potential exist for embarrassment to LBNL or DOE?
Initial Actions
Depending on the current level of risk, some immediate actions may be required. If other systems or accounts are at risk, then compromised systems should be physically disconnected from the network or otherwise prevented from posing a threat to other systems.
There are a number of increasingly aggressive methods for taking these initial actions:
• If the incident poses no immediate threat and analysis suggests that the activity may be legitimate, the involved parties are contacted for explanation.
• If the incident poses a threat, the Incident Handler will attempt to reach the designated contact(s) for the system and/or the appropriate liaison and have the system disconnected from the network.
• If the incident poses an immediate threat, the Incident Handler can take an immediate triage step of installing a filter at the DMZ, blocking systems involved in the incident from passing through the network perimeter.
• If network blocking is not sufficient to contain the threat, the Incident Handler may physically disconnect the system from the network with a note left on the console explaining the situation.
• If the system cannot be physically located and must be removed from network operation, the Incident Handler may enter the system (perhaps through a back door used by the hacker) and disable the system from network operation. In this case, a note will be broadcast to the console and all network interfaces will be configured down.
Every effort will be made to keep the system running until the Incident Handler can examine it. Except for triage steps necessary to prevent further damage, no corrective or investigatory actions should be taken by the system operator or administrator, except under the direction of the Incident Handler. This helps to preserve forensic evidence of unauthorized activity on the system.
Compromised System Follow-up - Owner Responsibilities
If a system has been compromised by an attack, such as a break-in or virus infection, a number of steps must be taken by the system administrator (or other responsible party) to insure that the compromise can not happen again.
Timeliness of Response
It is considered a security hole to allow a compromised system to remain uncorrected for long periods of time. An employee may notice a system in a corner that has been turned off for security purposes, and unwittingly use the system, being unaware of the security issues, thus putting LBNL at risk. To forestall these types of problems, the security issues must be addressed in a timely manner, as delineated below:
• The system must be corrected or decommissioned within 1 week of notification to the system administrator or other responsible party. The notification sent will include the expected correction date.
• In the event that the 1 week timeframe isn't met, the Incident Handler will secure the disk or other hardware as needed to prevent the compromised system from coming back on the network, and the division liaison will be informed. Upon obtaining a firm agreement on corrective action, including a timeline, the Incident Handler will relinquish the secured hardware to the responsible party for corrective action. If the system is not corrected in the agreed-upon timeframe, the hardware will again be secured, and the division management will be notified for corrective action.
• If the system is decommissioned, the administrator or responsible party must wipe, format, or otherwise permanently erase the data on the system disks, or render them unusable.
• At the CPPM's discretion, these regulations may be waived, or timeframes extended, as extraordinary circumstances warrant.
Root or Administrative Access Was Not Gained
If the Incident Handler is convinced root or administrative access was not gained:
• The compromised account(s) must be re-authenticated.
• The compromised account(s) must be checked for evidence of tampering.
• Any other systems that use the same credentials must be checked for evidence of tampering, and the credentials changed on those systems as well.
• If any “scanner” logs can be retrieved, they should be examined for local vulnerabilities, and any vulnerable remote sites should be notified.
Root or Administrative Privileges Were Gained
Some systems do not have a distinct boundary between user mode and root/administrative mode. In these cases, all compromises are considered root/administrative compromises.
If full and indisputable logs of all intruder activity can be recovered and all of the activity is fully understood, all observed changes must be restored to a reliable state. This can often be done when systems are attacked by viruses, or other automated attack tools. However, if the intruder had any chance to perform any activities that were either 1) not logged or 2) had questionable effects; the following steps must be performed:
• The entire system must be reinstalled from reliable media and all available security patches must be applied.
• If the method used to break in is known, steps must be taken to prevent it from being used again.
• All user accounts must be re-authenticated, that is, they must be disabled and require positive identification and a new password before being re-enabled.
• Appropriate examination of security sensitive system functionality must be performed. These steps may include, but are not limited to:
• File systems that weren't replaced during reinstallation should be scanned for any evidences of hacker manipulation, such as SUID files or obviously tampered-with user files.
• On Windows systems, the Registry must be examined for evidence of tampering.
• Trust relationships between the compromised system and other systems must be examined. Systems that trust the compromised systems may, after examination, be considered compromised, at or above the level of trust involved.
If any “sniffer” or “scanner” logs can be retrieved, all local compromised accounts should be disabled and any remote sites, which may have been compromised, should be notified. The system must not be placed back on the network until these steps have been taken and the Incident Handler has examined the system.
Other Actions
The Incident Handler may require other actions be performed. These may include, but are not limited to:
• If necessary, appropriate backups of the compromised system, or its log files may be requested for possible future forensic analysis, or evidentiary purposes. The Incident Handler may create forensic copies of the relevant media to facilitate investigation.
• Patching the system with the latest vendor-supplied, or other recommended security patches.
• Eliminating or reducing unnecessary network services.
• Installing and maintaining appropriate security software (e.g. Antivirus software)
• Forwarding log files to a central store for analysis.
• Successfully passing a network security scan.
• Attending appropriate cyber security training sessions.
Reconnection to Network
The system may not be connected to the network until the Incident Handler is satisfied that the system no longer represents a risk.
Compromised System Follow-up - CPP Responsibilities
Before the incident can be closed, the Incident Handler must perform the following actions.
• If applicable, a report is to be filed with the site that the attack came from, and with the Computer Incident Advisory Capability (CIAC). LBNL will comply with all applicable reporting regulations.
• If appropriate, information about how to secure systems against the same attack should be disseminated to the appropriate cybersecurity related mailing lists.
• If deemed necessary or prudent, a sitewide scan may be undertaken to find additional systems vulnerable to the same or similar attacks.
• A forensic analysis is to be performed, to ensure that all relevant information is known, so that appropriate countermeasures can be developed, or higher-level incident correlation can be made.
• An analysis is to be made of the cost of the incident, and this data recorded in LBNL's incident database.
• The Network Equipment Tracking System (NETS) is to be updated with any inaccuracies discovered during the course of the incident investigation.
CPP Incident Reporting
The primary priority for LBNL incident response will always be to protect LBNL cyber resources from harm. Executing protective measures will always take precedence over reporting.
Initial Reporting
The incident is to be reported as soon as possible to appropriate LBNL internal cyber-security mailing lists.
In addition, if there are any compromised accounts or systems, the incident is to be reported to the appropriate CPIC liaison(s) and the person(s) directly responsible for the compromised systems and/or accounts if known.
At this point, reporting outside of LBNL will be restricted to those sites or individuals who may be directly affected by the incident. This reporting is performed by the Incident Handler. In general, such reporting should not happen until initial actions have been taken to secure compromised systems and/or accounts.
Any electronic reports that reveal currently active passwords or easily guessable encrypted passwords should not be transmitted in clear-text. In most cases this password can be replaced by ellipses (…) for reporting purposes. If the data to be reported is of such a nature that unauthorized access to it could be damaging to LBNL, DOE, or to active investigations, such data may be encrypted to ensure secure communication
When collecting information on the incident, the following data may be collected, as appropriate to the incident:
• System(s) involved
• IP address(es)
• Platform (including release and patchlevel)
• Type of attack and identifiable characteristics of attack.
• Attacker Identity (if known)
• System owner(s) (or other responsible parties)
• Timeline of Incident (Dates and times if possible)
• Method of detection of attack
• Impact on attacked host
• Criticality of host
• Appropriate forensic evidence as determined by the Incident Handler.
LBNL/CIAC Reporting Procedures
LBNL will report the following types of incidents to CIAC:
• Root or Account compromises of systems
• Defacement of Web resources
• Malicious Code, including viruses, trojan horses, worms, or other code that are designed to cause significant damage will be reported under the following circumstances:
• The malicious code is new (previously unknown)
• The malicious code is known but has never been seen before (i.e. may be known, but is an initial sighting),
• The malicious code is known and has been seen before, but is reoccurring in an epidemic fashion.
• Probes and Reconnaissance Scans that succeed in scanning a major portion of LBNL's networks for critical services, such as DNS or repeated scans that seem to originate from the same source.
• Unauthorized access to LBNL compute resources will be reported, as will unsuccessful attempts at unauthorized access, if the attempt uses new techniques or is persistent.
• Denial of Service that affects a critical service, including, but not limited to, a primary E-mail server, a primary web server, or Internet web access will be reported.
Method of Reporting
LBNL will report to CIAC via e-mail or through the CIAC web page for reporting incidents. Incidents of a sensitive nature will be encrypted or otherwise relayed by non-public means. Note that LBNL does not work with classified information, does not have any facilities for storing or transmitting classified information, and does not maintain a staff of cleared individuals to handle classified information. Therefore, no incident reports coming from LBNL will be classified.
Contents of Report
LBNL will include as much summary information as is known. In the case of Unauthorized access, a summary of how access was attempted or gained will be included if that information is known. Detailed log files, if available, will be sent to CIAC upon request.
Timeliness of Reporting
The primary priority for LBNL incident response will always be to protect LBNL cyber resources from harm. As such, time will not be taken to report on the incident until the incident is deemed to be under control. CIAC will be notified of the incident in a timely manner once the incident is under control. Occasionally, incidents will extend beyond the original known information, as reported to CIAC. In such circumstances, ongoing status reports may be reported to CIAC, as appropriate.
If an incident is deemed an emergency due to its size, complexity, or impact (which may include embarrassment to the site or to DOE), the incident will be reported via the CIAC skypage number: 888-449-8369.
Reporting to Other Agencies
Any additional reporting will be made in accordance with appropriate DOE or LBNL guidelines. Currently, incidents are reported to a mailing list (cp-iwar@lbl.gov ), which submits copies to CIAC, Berkeley Site Office, CounterIntelligence, and the DOE Office of Inspector General's Technology Crimes Section. LBL cooperates fully with these organizations in the furtherance of any investigations on cyber incidents.
|
