The need for artificial intelligence systems that are not only capable of mastering complicated tasks but also of explaining their decisions has massively gained attention over the last years. This also seems to offer opportunities for further interconnecting different approaches to artificial intelligence, such as machine learning and knowledge representation. This work considers the task of learning knowledge bases from agent behavior, with a focus on human-readability, comprehensibility and applications in games. In this context, it will be presented how knowledge can be organized and processed on multiple levels of abstraction, allowing for efficient reasoning and revision. It will be investigated how learning agents can benefit from incorporating the approaches into their learning processes. Examples and applications are provided, e.g., in the context of general video game playing. The most essential approaches are implemented in the InteKRator toolbox and show potential for being applied in other domains (e.g., in medical informatics).