University of New Brunswick, Fredericton, New Brunswick, Canada e-mails: fshoeleh@unb.ca, mehdiyadollahi@unb.ca"/> University of Tehran, Tehran, Iran e-mail: asadpour@ut.ac.ir"/>
Search
  • All Journals
Advanced Search
Search articles by Title, Author, Keywords, DOI
Submit Manuscript
7.7CiteScore
2.0Impact Factor
Advanced Search
MAP  >  The Knowledge Engineering Review
2020 Volume 35
Article Contents
Next Previous
RESEARCH ARTICLE   Open Access    

Domain adaptation-based transfer learning using adversarial networks

More Information
  • Abstract: There is an implicit assumption in machine learning techniques that each new task has no relation to the tasks previously learned. Therefore, tasks are often addressed independently. However, in some domains, particularly reinforcement learning (RL), this assumption is often incorrect because tasks in the same or similar domain tend to be related. In other words, even though tasks are quite different in their specifics, they may have general similarities, such as shared skills, making them related. In this paper, a novel domain adaptation-based method using adversarial networks is proposed to do transfer learning in RL problems. Our proposed method incorporates skills previously learned from source task to speed up learning on a new target task by providing generalization not only within a task but also across different, but related tasks. The experimental results indicate the effectiveness of our method in dealing with RL problems.
  • 加载中
  • Abel , D., et al.2018. Policy and value transfer in lifelong reinforcement learning. In International Conference on Machine Learning, 20–29.

    Google Scholar

    Ammar , H. B., Eaton , E., Ruvolo , P. & Taylor , M.2014. Online multi-task learning for policy gradient methods. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), 1206–1214.

    Google Scholar

    Ammar , H. B., Eaton , E., Ruvolo , P. & Taylor , M. E.2015. Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. In Proceedings of AAAI.

    Google Scholar

    Ammar , H. B., Eaton , E., Taylor , M. E., et al.2014. An automated measure of MDP similarity for transfer in reinforcement learning. In Workshops at the Twenty-Eighth AAAI Conference on Artificial Intelligence.

    Google Scholar

    Ammar , H. B., et al.2012. Reinforcement learning transfer via sparse coding. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems,. 1. International Foundation for Autonomous Agents and Multiagent Systems, 383–390.

    Google Scholar

    Asadi , M. & Huber , M.2007. Effective control knowledge transfer through learning skill and representation hierarchies. In 20th International Joint Conference on Artificial Intelligence, ICML, 2054–2059.

    Google Scholar

    Asadi , M. & Huber , M.2015. A dynamic hierarchical task transfer in multiple robot explorations. In Proceedings on the International Conference on Artificial Intelligence (ICAI), 8, 22–27.

    Google Scholar

    Barreto , A., et al.2017. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, 4055–4065.

    Google Scholar

    Ben-David , S., Blitzer , J., Crammer , K., Kulesza , A., et al.2010. A theory of learning from different domains. Machine Learning79(1–2), 151–175.

    Google Scholar

    Ben-David , S., Blitzer , J., Crammer , K. & Pereira , F.2007. Analysis of representations for domain adaptation. In: Advances in Neural Information Processing Systems, 137–144.

    Google Scholar

    Bocsi , B., Csató , L. & Peters , J.2013. Alignment-based transfer learning for robot models. In The 2013 International Joint Conference on Neural Networks (IJCNN), 1–7. IEEE.

    Google Scholar

    Celiberto L. A., Jr., et al.2011. Using cases as heuristics in reinforcement learning: a transfer learning application. In: IJCAI Proceedings-International Joint Conference on Artificial Intelligence22(1), 1211.

    Google Scholar

    Cheng , Q., Wang , X. & Shen , L.2017. An autonomous inter-task mapping learning method via artificial neural network for transfer learning. In 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), 768–773. IEEE.

    Google Scholar

    Cheng , Q., Wang , X. & Shen , L.2019. A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research64, 645–703.

    Google Scholar

    Da Silva , F. L. & Reali Costa , A. H.2017. Towards zero-shot autonomous inter-task mapping through object-oriented task description. In: Workshop on Transfer in Reinforcement Learning (TiRL).

    Google Scholar

    Da Silva , F. L. & Reali Costa , A. H.2019. A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research64, 645–703.

    Google Scholar

    Da Silva , F. L., Glatt , R. & Reali Costa , A. H.2017. Simultaneously learning and advising in multiagent reinforcement learning. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, 1100–1108. International Foundation for Autonomous Agents and Multiagent Systems.

    Google Scholar

    Dabney , W. & Barto , A. G.2012. Adaptive step-size for online temporal difference learning. In Twenty-Sixth AAAI Conference on Artificial Intelligence.

    Google Scholar

    Fachantidis , A., et al.2011. Transfer learning via multiple inter-task mappings. In European Workshop on Reinforcement Learning, 225–236. Springer.

    Google Scholar

    Fachantidis , A., et al.2015. Transfer learning with probabilistic mapping selection. Adaptive Behavior23(1), 3–19.

    Google Scholar

    Ferns , N., Panangaden , P. & Precup , D.2011. Bisimulation metrics for continuous Markov decision processes. SIAM Journal on Computing40(6), 1662–1714.

    Google Scholar

    Ganin , Y. & Lempitsky , V.2014. Unsupervised domain adaptation by backpropagation. arXiv preprintarXiv:1409.7495.

    Google Scholar

    Ganin , Y. & Lempitsky , V. S.2015. Unsupervised domain adaptation by back-propagation. In ICML.

    Google Scholar

    Ganin , Y., Ustinova , E., et al.2016. Domain-adversarial training of neural networks. The Journal of Machine Learning Research17(1), 2096–2030.

    Google Scholar

    Goodfellow , I., et al.2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2672–2680.

    Google Scholar

    Hoffman , J., et al.2017. Simultaneous deep transfer across domains and tasks. In Domain Adaptation in Computer Vision Applications, 173–187. Springer.

    Google Scholar

    Konidaris , G. & Barto , A. G.2009. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in Neural Information Processing Systems, 1015–1023.

    Google Scholar

    Konidaris , G., Thomas , P., et al.2011. Value function approximation in reinforcement learning using the Fourier basis. In Proceedings of the Twenty-Fifth Conference on Artificial Intelligence, 380–385.

    Google Scholar

    Konidaris , G., et al.2012. Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research31(3), 360–375.

    Google Scholar

    Lazaric , A.2012. Transfer in reinforcement learning: a framework and a survey. Reinforcement Learning12, 143–173.

    Google Scholar

    Lazaric , A. & Restelli , M.2011. Transfer from multiple MDPs. In Advances in Neural Information Processing Systems, 1746–1754.

    Google Scholar

    Lazaric , A., Restelli , M. & Bonarini , A.2008. Transfer of samples in batch reinforcement learning. In: Proceedings of the 25th International Conference on Machine Learning – ICML 2008, pp. 544–551. ACM Press.

    Google Scholar

    Liu , M.-Y. & Tuzel , O.2016. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems, 469–477.

    Google Scholar

    Mahadevan , S. & Maggioni , M.2007. Proto-value functions: a Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research8, 2169–2231, 16.

    Google Scholar

    Moradi , P., et al.2012. Automatic skill acquisition in reinforcement learning using graph centrality measures. Intelligent Data Analysis16, 113–135.

    Google Scholar

    Puterman , M. L.2014. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons.

    Google Scholar

    Shoeleh , F. & Asadpour , M.2017. Graph based skill acquisition and transfer Learning for continuous reinforcement learning domains. Pattern Recognition Letters87, 104–116.

    Google Scholar

    Shoeleh , F. & Asadpour , M.2019. Skill based transfer learning with domain adaptation for continuous reinforcement learning domains. Applied Intelligence, 1–17.

    Google Scholar

    Spector , B. & Belongie , S.2018. Sample-effcient reinforcement learning through transfer and architectural priors. arXiv preprintarXiv:1801.02268.

    Google Scholar

    Sutton , R. S. S., Precup , D. & Singh , S.1999. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence112(1–2), 181–211.

    Google Scholar

    Taylor , M. E. & Stone , P.2009. Transfer learning for reinforcement learning domains: a survey. Journal of Machine Learning Research10, 1633–1685.

    Google Scholar

    Taylor , M. E. & Stone , P.2011. An introduction to intertask transfer for reinforcement learning. AI Magazine32(1), 15.

    Google Scholar

    Taylor , M. E., Stone , P. & Liu , Y.2007. Transfer learning via inter-task mappings for temporal difference learning. Journal of Machine Learning Research8, 2125–2167.

    Google Scholar

    Tzeng , E., et al.2017. Adversarial discriminative domain adaptation. Computer Vision and Pattern Recognition (CVPR)1(2), 4.

    Google Scholar

  • Cite this article

    Farzaneh Shoeleh, Mohammad Mehdi Yadollahi, Masoud Asadpour. 2020. Domain adaptation-based transfer learning using adversarial networks. The Knowledge Engineering Review 35(1), doi: 10.1017/S0269888920000107
    Farzaneh Shoeleh, Mohammad Mehdi Yadollahi, Masoud Asadpour. 2020. Domain adaptation-based transfer learning using adversarial networks. The Knowledge Engineering Review 35(1), doi: 10.1017/S0269888920000107
    shu
Download PDF

Article Metrics

Article views(55) PDF downloads(143)

Other Articles By Authors

RESEARCH ARTICLE   Open Access    

Domain adaptation-based transfer learning using adversarial networks

  • Received: 15 September 2019
  • Revised: 20 January 2020
  • Accepted: 21 January 2020
  • Published online: 26 February 2020
The Knowledge Engineering Review  35,  2020, 35(1)  |  Cite this article

Abstract: Abstract: There is an implicit assumption in machine learning techniques that each new task has no relation to the tasks previously learned. Therefore, tasks are often addressed independently. However, in some domains, particularly reinforcement learning (RL), this assumption is often incorrect because tasks in the same or similar domain tend to be related. In other words, even though tasks are quite different in their specifics, they may have general similarities, such as shared skills, making them related. In this paper, a novel domain adaptation-based method using adversarial networks is proposed to do transfer learning in RL problems. Our proposed method incorporates skills previously learned from source task to speed up learning on a new target task by providing generalization not only within a task but also across different, but related tasks. The experimental results indicate the effectiveness of our method in dealing with RL problems.

    HTML

    • https://github.com/ddtm/caffe/tree/grl

    Rights and permissions
    • © Cambridge University Press, 20202020Cambridge University Press
References (44)
  • About this article
    Cite this article
    Farzaneh Shoeleh, Mohammad Mehdi Yadollahi, Masoud Asadpour. 2020. Domain adaptation-based transfer learning using adversarial networks. The Knowledge Engineering Review 35(1), doi: 10.1017/S0269888920000107
    Farzaneh Shoeleh, Mohammad Mehdi Yadollahi, Masoud Asadpour. 2020. Domain adaptation-based transfer learning using adversarial networks. The Knowledge Engineering Review 35(1), doi: 10.1017/S0269888920000107
    shu

    • Catalog

      /

      DownLoad:  Full-Size Img  PowerPoint
      Return
      Return